Understanding self-supervised Learning Dynamics without Contrastive Pairs

Abstract

While contrastive approaches of self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point (positive pairs) and maximizing views from different data points (negative pairs), recent \emph{non-contrastive} SSL (e.g., BYOL and SimSiam) show remarkable performance {\it without} negative pairs, with an extra learnable predictor and a stop-gradient operation. A fundamental question arises: why do these methods not collapse into trivial representations? We answer this question via a simple theoretical study and propose a novel approach, DirectPred, that \emph{directly} sets the linear predictor based on the statistics of its inputs, without gradient training. On ImageNet, it performs comparably with more complex two-layer non-linear predictors that employ BatchNorm and outperforms a linear predictor by $2.5\%$ in 300-epoch training (and $5\%$ in 60-epoch). DirectPred is motivated by our theoretical study of the nonlinear learning dynamics of non-contrastive SSL in simple linear networks. Our study yields conceptual insights into how non-contrastive SSL methods learn, how they avoid representational collapse, and how multiple factors, like predictor networks, stop-gradients, exponential moving averages, and weight decay all come into play. Our simple theory recapitulates the results of real-world ablation studies in both STL-10 and ImageNet. Code is released https://github.com/facebookresearch/luckmatters/tree/master/ssl.

arXiv commentFix minor typo in Appendix

References in corpus (15)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Contrastive Multiview Coding
• Large Batch Training of Convolutional Networks
• Exploring Simple Siamese Representation Learning
• A mathematical theory of semantic development in deep neural networks
• Spurious Local Minima are Common in Two-Layer ReLU Neural Networks
• The Emergence of Spectral Universality in Deep Networks
• Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputs
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Width Provably Matters in Optimization for Deep Linear Neural Networks

Cited by in corpus (15)

• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Pre-training Molecular Graph Representation with 3D Geometry
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning
• Self-supervised Learning is More Robust to Dataset Imbalance
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Large-Scale Representation Learning on Graphs via Bootstrapping
• Targeted Supervised Contrastive Learning for Long-Tailed Recognition
• Revisiting consistency for semi-supervised semantic segmentation
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
• DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability
• On Feature Decorrelation in Self-Supervised Learning
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
• On the Regularization of Autoencoders