Active Contrastive Learning of Audio-Visual Video Representations

Abstract

Contrastive learning has been shown to produce generalizable representations of audio and visual data by maximizing the lower bound on the mutual information (MI) between different views of an instance. However, obtaining a tight lower bound requires a sample size exponential in MI and thus a large set of negative samples. We can incorporate more samples by building a large queue-based dictionary, but there are theoretical limits to performance improvements even with a large number of negative samples. We hypothesize that \textit{random negative sampling} leads to a highly redundant dictionary that results in suboptimal representations for downstream tasks. In this paper, we propose an active contrastive learning approach that builds an \textit{actively sampled} dictionary with diverse and informative items, which improves the quality of negative samples and improves performances on tasks where there is high mutual information in the data, e.g., video classification. Our model achieves state-of-the-art performance on challenging audio and visual downstream benchmarks including UCF101, HMDB51 and ESC50.\footnote{Code is available at: \url{https://github.com/yunyikristy/CM-ACC}}

References in corpus (12)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Learning deep representations by mutual information estimation and maximization
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Contrastive Multiview Coding
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• On Mutual Information in Contrastive Learning for Visual Representations

Cited by in corpus (9)

• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Contrastive Multi-Level Graph Neural Networks for Session-based Recommendation
• Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning
• Unsupervised Sound Localization via Iterative Contrastive Learning
• Training Efficiency and Robustness in Deep Learning
• V-SlowFast Network for Efficient Visual Sound Separation
• TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation
• MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing
• On Compositions of Transformations in Contrastive Self-Supervised Learning