Self-Supervised Learning for Videos: A Survey

Abstract

The remarkable success of deep learning in various domains relies on the availability of large-scale annotated datasets. However, obtaining annotations is expensive and requires great effort, which is especially challenging for videos. Moreover, the use of human-generated annotations leads to models with biased learning and poor domain generalization and robustness. As an alternative, self-supervised learning provides a way for representation learning which does not require annotations and has shown promise in both image and video domains. Different from the image domain, learning video representations are more challenging due to the temporal dimension, bringing in motion and other environmental dynamics. This also provides opportunities for video-exclusive ideas that advance self-supervised learning in the video and multimodal domain. In this survey, we provide a review of existing approaches on self-supervised learning focusing on the video domain. We summarize these methods into four different categories based on their learning objectives: 1) pretext tasks, 2) generative learning, 3) contrastive learning, and 4) cross-modal agreement. We further introduce the commonly used datasets, downstream evaluation tasks, insights into the limitations of existing works, and the potential future directions in this area.

arXiv commentACM CSUR (December 2022). Project Link: https://bit.ly/3Oimc7Q

References in corpus (41)

• Efficient Estimation of Word Representations in Vector Space
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Kinetics Human Action Video Dataset
• VoxCeleb2: Deep Speaker Recognition
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Self-supervised Learning: Generative or Contrastive
• Momentum Contrast for Unsupervised Visual Representation Learning
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Generating Videos with Scene Dynamics
• Lip Reading Sentences in the Wild
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Self-supervised Co-training for Video Representation Learning
• One Model To Learn Them All
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• A Closer Look at Spatiotemporal Convolutions for Action Recognition
• Self-Supervised MultiModal Versatile Networks
• Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
• UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
• TCLR: Temporal Contrastive Learning for Video Representation
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Improvements to context based self-supervised learning
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Robust Scene Text Recognition with Automatic Rectification
• The AVA-Kinetics Localized Human Actions Video Dataset
• Spatiotemporal Contrastive Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Support-set bottlenecks for video-text representation learning
• VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
• Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion
• Representation Learning with Video Deep InfoMax
• Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoders
• End-to-End Learning of Visual Representations from Uncurated Instructional Videos
• AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
• Temporally Coherent Embeddings for Self-Supervised Video Representation Learning

Cited by in corpus (8)

• Video Transformers: A Survey
• From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape
• Self-supervised visual learning in the low-data regime: a comparative evaluation
• Augmentation-aware Self-supervised Learning with Conditioned Projector
• A Hierarchical Framework with Spatio-Temporal Consistency Learning for Emergence Detection in Complex Adaptive Systems
• The ParlaSpeech Collection of Automatically Generated Speech and Text Datasets from Parliamentary Proceedings
• Leveraging Self-Supervised Learning for Fetal Cardiac Planes Classification using Ultrasound Scan Videos
• Advancing Video Self-Supervised Learning via Image Foundation Models