Self-supervised Learning: Generative or Contrastive

Abstract

Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.

24 pages, 19 figures

References in corpus (3)

• LINE: Large-scale Information Network Embedding
• Pre-trained Models for Natural Language Processing: A Survey
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training

Cited by in corpus (50)

• Deep Neural Networks and Tabular Data: A Survey
• Graph Self-Supervised Learning: A Survey
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning
• Self-Supervised Speech Representation Learning: A Review
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• Natural Language Processing for Smart Healthcare
• A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions
• AI and 6G into the Metaverse: Fundamentals, Challenges and Future Research Trends
• Self-Supervised Learning for Videos: A Survey
• Semantics-Guided Contrastive Network for Zero-Shot Object detection
• Spatial-Temporal Hypergraph Self-Supervised Learning for Crime Prediction
• Towards Label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis
• Online Self-Supervised Deep Learning for Intrusion Detection Systems
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Semantic-aware Dense Representation Learning for Remote Sensing Image Change Detection
• Distributed Traffic Synthesis and Classification in Edge Networks: A Federated Self-supervised Learning Approach
• Self-Supervised Learning for Data Scarcity in a Fatigue Damage Prognostic Problem
• ASTROMER: A transformer-based embedding for the representation of light curves
• Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images
• SVFAP: Self-supervised Video Facial Affect Perceiver
• When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generation
• MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation
• Diffusion Model-based Contrastive Learning for Human Activity Recognition
• Exploiting generative self-supervised learning for the assessment of biological images with lack of annotations: a COVID-19 case-study
• A Comprehensive Survey on Self-Supervised Learning for Recommendation
• Self-Supervised Variational Auto-Encoders
• Complementary Calibration: Boosting General Continual Learning with Collaborative Distillation and Self-Supervision
• Nucleus-aware Self-supervised Pretraining Using Unpaired Image-to-image Translation for Histopathology Images
• Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detection
• Scalable Imaginary Time Evolution with Neural Network Quantum States
• GenSelfDiff-HIS: Generative Self-Supervision Using Diffusion for Histopathological Image Segmentation
• CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection
• Self-supervised Learning of LiDAR 3D Point Clouds via 2D-3D Neural Calibration
• animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacoustics
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• Rotational Augmented Noise2Inverse for Low-dose Computed Tomography Reconstruction
• Generic Speech Enhancement with Self-Supervised Representation Space Loss
• Audio Enhancement for Computer Audition -- An Iterative Training Paradigm Using Sample Importance
• Edema Estimation From Facial Images Taken Before and After Dialysis via Contrastive Multi-Patient Pre-Training
• Toward task-driven satellite image super-resolution
• Enhancing Hyperspectral Image Prediction with Contrastive Learning in Low-Label Regime
• Towards Better Modeling with Missing Data: A Contrastive Learning-based Visual Analytics Perspective
• Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition
• Self-supervised Change Detection in Multi-view Remote Sensing Images
• FedCoSR: Personalized Federated Learning with Contrastive Shareable Representations for Label Heterogeneity in Non-IID Data
• Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning
• SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation
• Time-based Self-supervised Learning for Wireless Capsule Endoscopy