The Power of Contrast for Feature Learning: A Theoretical Analysis

Abstract

Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, theoretical understanding of the superiority of contrastive learning is still limited. In this paper, under linear representation settings, (i) we provably show that contrastive learning outperforms the standard autoencoders and generative adversarial networks, two classical generative unsupervised learning methods, for both feature recovery and in-domain downstream tasks; (ii) we also illustrate the impact of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning. We verify our theory with numerical experiments.

arXiv comment78 pages, accepted by JMLR

References in corpus (11)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Solving high-dimensional partial differential equations using deep learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Limit theorems for sample eigenvalues in a generalized spiked population model
• Supervised Contrastive Learning
• From Principal Subspaces to Principal Components with Linear Autoencoders
• Few-Shot Learning via Learning the Representation, Provably
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• A Selective Overview of Deep Learning
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss