Understanding Self-supervised Learning with Dual Deep Networks

Abstract

We propose a novel theoretical framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks (e.g., SimCLR). First, we prove that in each SGD update of SimCLR with various loss functions, including simple contrastive loss, soft Triplet loss and InfoNCE loss, the weights at each layer are updated by a \emph{covariance operator} that specifically amplifies initial random selectivities that vary across data samples but survive averages over data augmentations. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a \emph{hierarchical latent tree model} (HLTM) and prove that the hidden neurons of deep ReLU networks can learn the latent variables in HLTM, despite the fact that the network receives \emph{no direct supervision} from these unobserved latent variables. This leads to a provable emergence of hierarchical features through the amplification of initially random selectivities through contrastive SSL. Extensive numerical studies justify our theoretical findings. Code is released in https://github.com/facebookresearch/luckmatters/tree/master/ssl.

References in corpus (24)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Exploring Simple Siamese Representation Learning
• On Exact Computation with an Infinitely Wide Neural Net
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• A mathematical theory of semantic development in deep neural networks
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• BYOL works even without batch statistics
• Contrastive learning, multi-view redundancy, and linear models
• Self-Supervised Representations Improve End-to-End Speech Translation
• Student Specialization in Deep ReLU Networks With Finite Width and Input Dimension

Cited by in corpus (16)

• On the Opportunities and Risks of Foundation Models
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Hard Negative Mixing for Contrastive Learning
• Intriguing Properties of Contrastive Losses
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Whitening for Self-Supervised Representation Learning
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• BYOL works even without batch statistics
• TriBYOL: Triplet BYOL for Self-Supervised Representation Learning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning
• Self-supervised Graph Neural Networks without explicit negative sampling
• Self-Supervised Learning for Gastritis Detection with Gastric X-ray Images
• Improving Few-Shot Learning with Auxiliary Self-Supervised Pretext Tasks
• Understanding the Role of Self-Supervised Learning in Out-of-Distribution Detection Task
• Inverse Problems Leveraging Pre-trained Contrastive Representations