Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Abstract

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further finetuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.

arXiv commentNote: Changed name - added experiments

Cited by in corpus (43)

• Transformers in Vision: A Survey
• Human Action Recognition from Various Data Modalities: A Review
• Multi-task Self-Supervised Learning for Human Activity Detection
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Self-supervised Co-training for Video Representation Learning
• Hard Negative Mixing for Contrastive Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Self-Supervised MultiModal Versatile Networks
• Audiovisual SlowFast Networks for Video Recognition
• Self-Supervised Learning for Videos: A Survey
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Space-Time Correspondence as a Contrastive Random Walk
• Spatiotemporal Contrastive Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Video Representation Learning with Visual Tempo Consistency
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• Active Contrastive Learning of Audio-Visual Video Representations
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
• Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds
• LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
• AVA-AVD: Audio-Visual Speaker Diarization in the Wild
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• What Makes Training Multi-Modal Classification Networks Hard?
• Improving colonoscopy lesion classification using semi-supervised deep learning
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Learning Video Representations from Textual Web Supervision
• Boosting Discriminative Visual Representation Learning with Scenario-Agnostic Mixup
• Automatic audiovisual synchronisation for ultrasound tongue imaging
• Pretext-Contrastive Learning: Toward Good Practices in Self-supervised Video Representation Leaning
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
• Back to the Future: Cycle Encoding Prediction for Self-supervised Contrastive Video Representation Learning
• Learning Audio-Visual Dereverberation
• Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization
• Recursive Visual Sound Separation Using Minus-Plus Net
• Cross-modal Learning for Multi-modal Video Categorization
• Contrastive Learning of Global-Local Video Representations
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing
• On Compositions of Transformations in Contrastive Self-Supervised Learning