Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
arXiv:2011.10944
Abstract
Recently, a newly proposed self-supervised framework Bootstrap Your Own Latent (BYOL) seriously challenges the necessity of negative samples in contrastive learning frameworks. BYOL works like a charm despite the fact that it discards the negative samples completely and there is no measure to prevent collapse in its training objective. In this paper, we suggest understanding BYOL from the view of our proposed interpretable self-supervised learning framework, Run Away From your Teacher (RAFT). RAFT optimizes two objectives at the same time: (i) aligning two views of the same data to similar representations and (ii) running away from the model's Mean Teacher (MT, the exponential moving average of the history models) instead of BYOL's running towards it. The second term of RAFT explicitly prevents the representation collapse and thus makes RAFT a more conceptually reliable framework. We provide basic benchmarks of RAFT on CIFAR10 to validate the effectiveness of our method. Furthermore, we prove that BYOL is equivalent to RAFT under certain conditions, providing solid reasoning for BYOL's counter-intuitive success.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- A Simple Framework for Contrastive Learning of Visual Representations
- Playing Atari with Deep Reinforcement Learning
- Bootstrap your own latent: A new approach to self-supervised Learning
- Improved Baselines with Momentum Contrastive Learning
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Temporal Ensembling for Semi-Supervised Learning
- Contrastive Multiview Coding
- Born Again Neural Networks
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- On Mutual Information Maximization for Representation Learning
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
- Self-Distillation Amplifies Regularization in Hilbert Space