Self-Supervised Representation Learning: Introduction, Advances and Challenges

Abstract

Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets, thus alleviating the annotation bottleneck that is one of the main barriers to practical deployment of deep learning today. These methods have advanced rapidly in recent years, with their efficacy approaching and sometimes surpassing fully supervised pre-training alternatives across a variety of data modalities including image, video, sound, text and graphs. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data. We further discuss practical considerations including workflows, representation transferability, and compute cost. Finally, we survey the major open challenges in the field that provide fertile ground for future work.

References in corpus (21)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• Bootstrap your own latent: A new approach to self-supervised Learning
• The Kinetics Human Action Video Dataset
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Cross-lingual Language Model Pretraining
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• What Makes for Good Views for Contrastive Learning?
• Predicting multicellular function through multi-layer tissue networks
• Rethinking Pre-training and Self-training
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
• Self-supervised Co-training for Video Representation Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Self-Supervised MultiModal Versatile Networks
• Clustering Stability: An Overview
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• 3D Self-Supervised Methods for Medical Imaging
• Subject-Aware Contrastive Learning for Biosignals
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

Cited by in corpus (25)

• Self-Supervised Speech Representation Learning: A Review
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Deep Learning Based Single Sample Per Person Face Recognition: A Survey
• DAS-N2N: Machine learning Distributed Acoustic Sensing (DAS) signal denoising without clean data
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• A Generic Fundus Image Enhancement Network Boosted by Frequency Self-supervised Representation Learning
• Detecting Intentional AIS Shutdown in Open Sea Maritime Surveillance Using Self-Supervised Deep Learning
• Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
• Learning image representations for anomaly detection: application to discovery of histological alterations in drug development
• Controllable Data Generation by Deep Learning: A Review
• MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
• GraSS: Contrastive Learning with Gradient Guided Sampling Strategy for Remote Sensing Image Semantic Segmentation
• A critical review of methods and challenges in large language models
• A General Framework for Generative Self-supervised Learning in Non-invasive Estimation of Physiological Parameters Using Photoplethysmography
• Robust and Explainable Framework to Address Data Scarcity in Diagnostic Imaging
• Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation
• An efficient unsupervised classification model for galaxy morphology: Voting clustering based on coding from ConvNeXt large model
• Leveraging Self-Supervised Learning for Scene Classification in Child Sexual Abuse Imagery
• Wearable data from subjects playing Super Mario, sitting university exams, or performing physical exercise help detect acute mood episodes via self-supervised learning
• SS-CPGAN: Self-Supervised Cut-and-Pasting Generative Adversarial Network for Object Segmentation
• SIGNL: A Label-Efficient Audio Deepfake Detection System via Spectral-Temporal Graph Non-Contrastive Learning
• Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning
• Exploring internal representation of self-supervised networks: few-shot learning abilities and comparison with human semantics and recognition of objects