Predicting What You Already Know Helps: Provable Self-Supervised Learning

Abstract

Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations. These pretext tasks are created solely using the input features, such as predicting a missing image patch, recovering the color channels of an image from context, or predicting missing words in text; yet predicting this \textit{known} information helps in learning representations effective for downstream prediction tasks. We posit a mechanism exploiting the statistical connections between certain {\em reconstruction-based} pretext tasks that guarantee to learn a good representation. Formally, we quantify how the approximate independence between the components of the pretext task (conditional on the label and latent variables) allows us to learn representations that can solve the downstream task by just training a linear layer on top of the learned representation. We prove the linear layer yields small approximation error even for complex ground truth function class and will drastically reduce labeled sample complexity. Next, we show a simple modification of our method leads to nonlinear CCA, analogous to the popular SimSiam algorithm, and show similar guarantees for nonlinear CCA.

arXiv commentNeurIPS 2021

References in corpus (18)

• Distributed Representations of Words and Phrases and their Compositionality
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Learning deep representations by mutual information estimation and maximization
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• An efficient framework for learning sentence representations
• Kernel dimension reduction in regression
• On Mutual Information Maximization for Representation Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• Random design analysis of ridge regression
• Few-Shot Learning via Learning the Representation, Provably
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Representation Learning via Invariant Causal Mechanisms
• Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
• Contrastive estimation reveals topic posterior information to linear models
• Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs

Cited by in corpus (31)

• On the Opportunities and Risks of Foundation Models
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• Hard Negative Mixing for Contrastive Learning
• Self-Training: A Survey
• Learning and Evaluating Representations for Deep One-class Classification
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• Self-supervised Learning is More Robust to Dataset Imbalance
• Intriguing Properties of Contrastive Losses
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Few-Shot Learning via Learning the Representation, Provably
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Understanding Self-supervised Learning with Dual Deep Networks
• Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Towards the Generalization of Contrastive Self-Supervised Learning
• Contrastive learning, multi-view redundancy, and linear models
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
• Self-supervised Learning from a Multi-view Perspective
• Investigating the Role of Negatives in Contrastive Representation Learning
• For self-supervised learning, Rationality implies generalization, provably
• How Fine-Tuning Allows for Effective Meta-Learning
• Conditional independence for pretext task selection in Self-supervised speech representation learning
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
• The Power of Contrast for Feature Learning: A Theoretical Analysis
• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches
• Self-training Converts Weak Learners to Strong Learners in Mixture Models
• On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• On a Benefit of Mask Language Modeling: Robustness to Simplicity Bias
• Pretext Tasks selection for multitask self-supervised speech representation learning