Self-Supervised MultiModal Versatile Networks

Abstract

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, AudioSet and ESC-50 when compared to previous self-supervised work. Our models are publicly available.

arXiv commentTo appear in the Thirty-Fourth Annual Conference on Neural Information Processing Systems (NeurIPS 2020)

References in corpus (24)

• Scikit-learn: Machine Learning in Python
• Efficient Estimation of Word Representations in Vector Space
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Learning deep representations by mutual information estimation and maximization
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• A Short Note about Kinetics-600
• One Model To Learn Them All
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Learning Video Representations using Contrastive Bidirectional Transformer
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Spatiotemporal Contrastive Video Representation Learning
• PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition
• See, Hear, and Read: Deep Aligned Representations

Cited by in corpus (28)

• Learning Transferable Visual Models From Natural Language Supervision
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Florence: A New Foundation Model for Computer Vision
• Self-Supervised Learning for Videos: A Survey
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• A survey of multimodal deep generative models
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• MERLOT: Multimodal Neural Script Knowledge Models
• Support-set bottlenecks for video-text representation learning
• TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification
• Multi-Format Contrastive Learning of Audio Representations
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding
• Multimodal neural networks better explain multivoxel patterns in the hippocampus
• Improving Speech Translation by Cross-Modal Multi-Grained Contrastive Learning
• Multimodal Contrastive Training for Visual Representation Learning
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
• Semantic Role Aware Correlation Transformer for Text to Video Retrieval
• VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
• ModSelect: Automatic Modality Selection for Synthetic-to-Real Domain Generalization
• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons
• ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning
• MM-Pyramid: Multimodal Pyramid Attentional Network for Audio-Visual Event Localization and Video Parsing