Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
arXiv:2101.07525
Abstract
In this paper, we present a novel approach, Momentum Teacher, for student-teacher based self-supervised learning. The approach performs momentum update on both network weights and batch normalization (BN) statistics. The teacher's weight is a momentum update of the student, and the teacher's BN statistics is a momentum update of those in history. The Momentum Teacher is simple and efficient. It can achieve the state of the art results (74.5\%) under ImageNet linear evaluation protocol using small-batch size(\eg, 128), without requiring large-batch training on special hardware like TPU or inefficient across GPU operation (\eg, shuffling BN, synced BN). Our implementation and pre-trained models will be given on GitHub\footnote{https://github.com/zengarden/momentum2-teacher}.
arXiv comment11 pages, Tech report
References in corpus (15)
- A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap your own latent: A new approach to self-supervised Learning
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Improved Baselines with Momentum Contrastive Learning
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Unsupervised Data Augmentation for Consistency Training
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Contrastive Multiview Coding
- Large Batch Training of Convolutional Networks
- Prototypical Contrastive Learning of Unsupervised Representations
- Exploring Simple Siamese Representation Learning
- Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization