Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
arXiv:2104.10955
Abstract
Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task-relevant semantics, which facilitates pulling together representations across modalities by compositional contrastive learning. We establish a new, comprehensive multi-modal distillation benchmark on three video datasets: UCF101, ActivityNet, and VGGSound. Moreover, we demonstrate that our model significantly outperforms a variety of existing knowledge distillation methods in transferring audio-visual knowledge to improve video representation learning. Code is released here: https://github.com/yanbeic/CCL.
Accepted to CVPR2021
References in corpus (6)
- Distilling the Knowledge in a Neural Network
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Learning Robust Representations via Multi-View Information Bottleneck