Self-Supervised Learning by Cross-Modal Audio-Video Clustering

Abstract

Visual and audio modalities are highly correlated, yet they contain different information. Their strong correlation makes it possible to predict the semantics of one from the other with good accuracy. Their intrinsic differences make cross-modal prediction a potentially more rewarding pretext task for self-supervised learning of video and audio representations compared to within-modality learning. Based on this intuition, we propose Cross-Modal Deep Clustering (XDC), a novel self-supervised method that leverages unsupervised clustering in one modality (e.g., audio) as a supervisory signal for the other modality (e.g., video). This cross-modal supervision helps XDC utilize the semantic correlation and the differences between the two modalities. Our experiments show that XDC outperforms single-modality clustering and other multi-modal variants. XDC achieves state-of-the-art accuracy among self-supervised methods on multiple video and audio benchmarks. Most importantly, our video model pretrained on large-scale unlabeled data significantly outperforms the same model pretrained with full-supervision on ImageNet and Kinetics for action recognition on HMDB51 and UCF101. To the best of our knowledge, XDC is the first self-supervised learning method that outperforms large-scale fully-supervised pretraining for action recognition on the same architecture.

Accepted to NeurIPS 2020 (spotlight presentation)

References in corpus (10)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Unsupervised Learning of Video Representations using LSTMs
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Learning Video Representations using Contrastive Bidirectional Transformer
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Learning visual groups from co-occurrences in space and time
• See, Hear, and Read: Deep Aligned Representations
• Self-supervised Learning for Video Correspondence Flow

Cited by in corpus (42)

• Human Action Recognition from Various Data Modalities: A Review
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Self-supervised Co-training for Video Representation Learning
• Self-Supervised MultiModal Versatile Networks
• TCLR: Temporal Contrastive Learning for Video Representation
• Self-Supervised Learning for Videos: A Survey
• What Should Not Be Contrastive in Contrastive Learning
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Are we done with ImageNet?
• Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching
• Video Representation Learning with Visual Tempo Consistency
• Active Contrastive Learning of Audio-Visual Video Representations
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Understanding Self-supervised Learning with Dual Deep Networks
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• Multi-Format Contrastive Learning of Audio Representations
• Video Understanding as Machine Translation
• A Survey of Video-based Action Quality Assessment
• Learning Representations from Audio-Visual Spatial Alignment
• Dual Contrastive Learning for Spatio-temporal Representation
• Learning Video Representations from Textual Web Supervision
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
• Self-supervised Video Representation Learning by Context and Motion Decoupling
• Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Click-through Rate Prediction with Auto-Quantized Contrastive Learning
• Visually Guided Sound Source Separation using Cascaded Opponent Filter Network
• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Contrastive Learning of Global-Local Video Representations
• Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Self-Supervised Video Representation Learning by Video Incoherence Detection
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency