A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Abstract

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

19 pages, 5 figures

Cited by in corpus (61)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Pre-trained Models for Natural Language Processing: A Survey
• Recent advances and clinical applications of deep learning in medical image analysis
• Contrastive Representation Learning: A Framework and Review
• Prototypical Contrastive Learning of Unsupervised Representations
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Hard Negative Mixing for Contrastive Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Learning Representation for Clustering via Prototype Scattering and Positive Sampling
• DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
• Learning and Evaluating Representations for Deep One-class Classification
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• A Mutual Information Maximization Perspective of Language Representation Learning
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Few-Shot Learning via Learning the Representation, Provably
• Graph Pooling via Coarsened Graph Infomax
• Representation Learning via Invariant Causal Mechanisms
• Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity
• Provable Guarantees for Gradient-Based Meta-Learning
• Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding
• Graph Foundation Models: Concepts, Opportunities and Challenges
• Understanding the Behaviour of Contrastive Loss
• Incremental False Negative Detection for Contrastive Learning
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning
• Open-Set Hypothesis Transfer with Semantic Consistency
• On Learning Universal Representations Across Languages
• Investigating the Role of Negatives in Contrastive Representation Learning
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• Supervised Contrastive Learning for Accented Speech Recognition
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• For self-supervised learning, Rationality implies generalization, provably
• Calibrating and Improving Graph Contrastive Learning
• PAC-Bayesian Contrastive Unsupervised Representation Learning
• Strength from Weakness: Fast Learning Using Weak Supervision
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Conditional independence for pretext task selection in Self-supervised speech representation learning
• Negative-Free Self-Supervised Gaussian Embedding of Graphs
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Conditional Alignment and Uniformity for Contrastive Learning with Continuous Proxy Labels
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Theoretical bounds on estimation error for meta-learning
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• Self-Supervised Contextual Bandits in Computer Vision
• Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
• Contrastive learning of strong-mixing continuous-time stochastic processes
• Unsupervised Abstract Reasoning for Raven's Problem Matrices
• About contrastive unsupervised representation learning for classification and its convergence
• Boosting Few-Shot Classification with View-Learnable Contrastive Learning