Contrastive Representation Learning: A Framework and Review

Abstract

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

28 pages, 9 figures, update with the accepted version in IEEE Access

References in corpus (51)

• Efficient Estimation of Word Representations in Vector Space
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Auto-Encoding Variational Bayes
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Semi-Supervised Classification with Graph Convolutional Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Deep Residual Learning for Image Recognition
• Neural Message Passing for Quantum Chemistry
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Bootstrap your own latent: A new approach to self-supervised Learning
• Categorical Reparameterization with Gumbel-Softmax
• In Defense of the Triplet Loss for Person Re-Identification
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Scaling Laws for Neural Language Models
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Exploring the Limits of Language Modeling
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• An efficient framework for learning sentence representations
• Contrastive Multi-View Representation Learning on Graphs
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• On Mutual Information Maximization for Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• On Variational Bounds of Mutual Information
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Contrastive Learning for Unpaired Image-to-Image Translation
• Contrastive Code Representation Learning
• What Should Not Be Contrastive in Contrastive Learning
• What makes instance discrimination good for transfer learning?
• A Mutual Information Maximization Perspective of Language Representation Learning
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Contrastive Learning of Structured World Models
• Decoupling Representation Learning from Reinforcement Learning
• Whitening for Self-Supervised Representation Learning
• Neural Predictive Belief Representations
• Watching the World Go By: Representation Learning from Unlabeled Videos
• LoCo: Local Contrastive Representation Learning
• Delving into Inter-Image Invariance for Unsupervised Visual Representations
• Online Object Representations with Contrastive Learning
• Representation Learning with Video Deep InfoMax
• COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning

Cited by in corpus (62)

• Self-supervised learning methods and applications in medical imaging analysis: A survey
• CARLA: Self-supervised Contrastive Representation Learning for Time Series Anomaly Detection
• ECG Biometric Authentication Using Self-Supervised Learning for IoT Edge Sensors
• NORESQA: A Framework for Speech Quality Assessment using Non-Matching References
• Evaluating Contrastive Models for Instance-based Image Retrieval
• Enriched Music Representations with Multiple Cross-modal Contrastive Learning
• MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
• LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning
• Supervised Contrastive Learning for Affect Modelling
• Clinically Labeled Contrastive Learning for OCT Biomarker Classification
• GeoCLR: Georeference Contrastive Learning for Efficient Seafloor Image Interpretation
• NewsEmbed: Modeling News through Pre-trained Document Representations
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Collaborative-Enhanced Prediction of Spending on Newly Downloaded Mobile Games under Consumption Uncertainty
• Disambiguated Node Classification with Graph Neural Networks
• End-to-End Simultaneous Speech Translation with Differentiable Segmentation
• Investigating the Role of Negatives in Contrastive Representation Learning
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• Generalizable Pancreas Segmentation via a Dual Self-Supervised Learning Framework
• Learning the Universe: Cosmological and Astrophysical Parameter Inference with Galaxy Luminosity Functions and Colours
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Contrastive Learning for Predicting Cancer Prognosis Using Gene Expression Values
• ContrastNER: Contrastive-based Prompt Tuning for Few-shot NER
• Goals as Reward-Producing Programs
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• A Generic Self-Supervised Framework of Learning Invariant Discriminative Features
• Stimulate the Potential of Robots via Competition
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• Heterogeneous Subgraph Network with Prompt Learning for Interpretable Depression Detection on Social Media
• Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification
• A Novel Transformer-Based Self-Supervised Learning Method to Enhance Photoplethysmogram Signal Artifact Detection
• Style Feature Extraction Using Contrastive Conditioned Variational Autoencoders with Mutual Information Constraints
• Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches
• Triplet Contrastive Learning for Brain Tumor Classification
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Learning Generalized Hybrid Proximity Representation for Image Recognition
• Edema Estimation From Facial Images Taken Before and After Dialysis via Contrastive Multi-Patient Pre-Training
• Exploring Galaxy Properties of eCALIFA with Contrastive Learning
• Enhancing Hyperspectral Image Prediction with Contrastive Learning in Low-Label Regime
• GOTPR: General Outdoor Text-based Place Recognition Using Scene Graph Retrieval with OpenStreetMap
• Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Data
• Deep-Learned Event Variables for Collider Phenomenology
• Sensor Data Augmentation by Resampling for Contrastive Learning in Human Activity Recognition
• GeomCA: Geometric Evaluation of Data Representations
• RMT-BVQA: Recurrent Memory Transformer-based Blind Video Quality Assessment for Enhanced Video Content
• Contrastive Fine-tuning Improves Robustness for Neural Rankers
• Dissecting Supervised Contrastive Learning
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Malfustection: Obfuscated Malware Detection and Malware Classification with Data Shortage by Combining Semi-Supervised and Contrastive Learning
• Neuro-Symbolic Contrastive Learning for Cross-domain Inference
• Cryo-forum: A framework for orientation recovery with uncertainty measure with the application in cryo-EM image analysis
• Large Models for Battery Prognostics and Health Management: A Review and Future Roadmap
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
• Contrastive Learning for Aspect Representation towards Explainable Recommendation
• Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
• CoRAST: Towards Foundation Model-Powered Correlated Data Analysis in Resource-Constrained CPS and IoT
• Tight PAC-Bayesian Risk Certificates for Contrastive Learning
• Generative Adversarial Learning via Kernel Density Discrimination