A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Abstract

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

19 pages, 5 figures

Cited by in corpus (23)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Contrastive Representation Learning: A Framework and Review
• Learning and Evaluating Representations for Deep One-class Classification
• A Mutual Information Maximization Perspective of Language Representation Learning
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Graph Pooling via Coarsened Graph Infomax
• Provable Guarantees for Gradient-Based Meta-Learning
• Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding
• Understanding the Behaviour of Contrastive Loss
• Few-shot Visual Reasoning with Meta-analogical Contrastive Learning
• Investigating the Role of Negatives in Contrastive Representation Learning
• Supervised Contrastive Learning for Accented Speech Recognition
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• Strength from Weakness: Fast Learning Using Weak Supervision
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• CoSQA: 20,000+ Web Queries for Code Search and Question Answering
• Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification
• Boosting Few-Shot Classification with View-Learnable Contrastive Learning
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization
• Unsupervised Abstract Reasoning for Raven's Problem Matrices
• About contrastive unsupervised representation learning for classification and its convergence
• Contrastive learning of strong-mixing continuous-time stochastic processes