Self-Supervised Representation Learning: Introduction, Advances and Challenges
arXiv:2110.09327 · doi:10.1109/MSP.2021.3134634
Abstract
Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work.
References in corpus (36)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distributed Representations of Words and Phrases and their Compositionality
- Representation Learning with Contrastive Predictive Coding
- Learning Transferable Visual Models From Natural Language Supervision
- Bootstrap your own latent: A new approach to self-supervised Learning
- The Kinetics Human Action Video Dataset
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
- Cross-lingual Language Model Pretraining
- YFCC100M: The New Data in Multimedia Research
- Unsupervised Representation Learning by Predicting Image Rotations
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- What Makes for Good Views for Contrastive Learning?
- Predicting multicellular function through multi-layer tissue networks
- Libri-Light: A Benchmark for ASR with Limited or No Supervision
- Deep Anomaly Detection Using Geometric Transformations
- Rethinking Pre-training and Self-training
- Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
- Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
- Self-supervised Co-training for Video Representation Learning
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Unsupervised Scalable Representation Learning for Multivariate Time Series
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- Self-Supervised MultiModal Versatile Networks
- Strategies for Pre-training Graph Neural Networks
- Self-supervised Learning for ECG-based Emotion Recognition
- Clustering Stability: An Overview
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
- 3D Self-Supervised Methods for Medical Imaging
- Subject-Aware Contrastive Learning for Biosignals
- GPT-GNN: Generative Pre-Training of Graph Neural Networks
- Predicting What You Already Know Helps: Provable Self-Supervised Learning
- A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
Cited by in corpus (25)
- Self-Supervised Speech Representation Learning: A Review
- Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
- Deep Learning Based Single Sample Per Person Face Recognition: A Survey
- DAS-N2N: Machine learning Distributed Acoustic Sensing (DAS) signal denoising without clean data
- A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
- A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
- Detecting Intentional AIS Shutdown in Open Sea Maritime Surveillance Using Self-Supervised Deep Learning
- Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis
- A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
- Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
- Learning image representations for anomaly detection: application to discovery of histological alterations in drug development
- Controllable Data Generation by Deep Learning: A Review
- MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
- GraSS: Contrastive Learning with Gradient Guided Sampling Strategy for Remote Sensing Image Semantic Segmentation
- A critical review of methods and challenges in large language models
- A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography
- Robust and Explainable Framework to Address Data Scarcity in Diagnostic Imaging
- Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation
- An efficient unsupervised classification model for galaxy morphology: Voting clustering based on coding from ConvNeXt large model
- Leveraging Self-Supervised Learning for Scene Classification in Child Sexual Abuse Imagery
- Wearable data from subjects playing Super Mario, sitting university exams, or performing physical exercise help detect acute mood episodes via self-supervised learning
- SS-CPGAN: Self-Supervised Cut-and-Pasting Generative Adversarial Network for Object Segmentation
- SIGNL: A Label-Efficient Audio Deepfake Detection System via Spectral-Temporal Graph Non-Contrastive Learning
- Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning
- Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects