Distilling Audio-Visual Knowledge by Compositional Contrastive Learning
arXiv:2104.10955
Abstract
Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task-relevant semantics, which facilitates pulling together representations across modalities by compositional contrastive learning. We establish a new, comprehensive multi-modal distillation benchmark on three video datasets: UCF101, ActivityNet, and VGGSound. Moreover, we demonstrate that our model significantly outperforms a variety of existing knowledge distillation methods in transferring audio-visual knowledge to improve video representation learning. Code is released here: https://github.com/yanbeic/CCL.
arXiv commentAccepted to CVPR2021
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Improved Baselines with Momentum Contrastive Learning
- Learning deep representations by mutual information estimation and maximization
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Deep Audio-Visual Speech Recognition
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Use What You Have: Video Retrieval Using Representations From Collaborative Experts
- Learning Robust Representations via Multi-View Information Bottleneck
- PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition