Contrastive Representation Learning: A Framework and Review

Abstract

Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and unifies many different contrastive learning methods. We also provide a taxonomy for each of the components of contrastive learning in order to summarise it and distinguish it from other forms of machine learning. We then discuss the inductive biases which are present in any contrastive learning system and we analyse our framework under different views from various sub-fields of Machine Learning. Examples of how contrastive learning has been applied in computer vision, natural language processing, audio processing, and others, as well as in Reinforcement Learning are also presented. Finally, we discuss the challenges and some of the most promising future research directions ahead.

28 pages, 9 figures, update with the accepted version in IEEE Access

References in corpus (17)

• Semi-Supervised Classification with Graph Convolutional Networks
• Bootstrap your own latent: A new approach to self-supervised Learning
• Scaling Laws for Neural Language Models
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multi-View Representation Learning on Graphs
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• On Variational Bounds of Mutual Information
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Contrastive Learning for Unpaired Image-to-Image Translation
• A Mutual Information Maximization Perspective of Language Representation Learning
• Contrastive Learning of Structured World Models
• Online Object Representations with Contrastive Learning
• Representation Learning with Video Deep InfoMax
• COALA: Co-Aligned Autoencoders for Learning Semantically Enriched Audio Representations
• CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning

Cited by in corpus (7)

• NORESQA: A Framework for Speech Quality Assessment using Non-Matching References
• Enriched Music Representations with Multiple Cross-modal Contrastive Learning
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• Set Representation Learning with Generalized Sliced-Wasserstein Embeddings
• FROST: Faster and more Robust One-shot Semi-supervised Training
• Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
• Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast