Contrastive learning, multi-view redundancy, and linear models
arXiv:2008.10150
Abstract
Self-supervised learning is an empirically successful approach to unsupervised learning based on creating artificial supervised learning problems. A popular self-supervised approach to representation learning is contrastive learning, which leverages naturally occurring pairs of similar and dissimilar data points, or multiple views of the same data. This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available. The main result is that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.
References in corpus (12)
- The information bottleneck method
- Learning deep representations by mutual information estimation and maximization
- Contrastive Multiview Coding
- Identifiability of parameters in latent structure models with many observed variables
- An efficient framework for learning sentence representations
- On Mutual Information Maximization for Representation Learning
- Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
- A Spectral Algorithm for Learning Hidden Markov Models
- Provably efficient RL with Rich Observations via Latent State Decoding
- Predicting What You Already Know Helps: Provable Self-Supervised Learning
- Contrastive estimation reveals topic posterior information to linear models
- Dirichlet draws are sparse with high probability
Cited by in corpus (5)
- Learning and Evaluating Representations for Deep One-class Classification
- Intriguing Properties of Contrastive Losses
- Understanding Self-supervised Learning with Dual Deep Networks
- Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
- Towards Demystifying Representation Learning with Non-contrastive Self-supervision