Contrastive Learning Inverts the Data Generating Process

Abstract

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

Presented at ICML 2021. The first three authors, as well as the last two authors, contributed equally. Code is available at https://brendel-group.github.io/cl-ica

References in corpus (9)

• A Simple Framework for Contrastive Learning of Visual Representations
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Weakly-Supervised Disentanglement Without Compromises
• Contrastive Learning with Hard Negative Samples
• Debiased Contrastive Learning
• On Linear Identifiability of Learned Representations

Cited by in corpus (5)

• On the Opportunities and Risks of Foundation Models
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Self-Supervised Learning with Kernel Dependence Maximization
• Unsupervised Learning of Compositional Energy Concepts
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning