Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning

Abstract

In this paper, we present a novel approach, Momentum${}^2$ Teacher, for student-teacher based self-supervised learning. The approach performs momentum update on both network weights and batch normalization (BN) statistics. The teacher's weight is a momentum update of the student, and the teacher's BN statistics is a momentum update of those in history. The Momentum${}^2$ Teacher is simple and efficient. It can achieve the state of the art results (74.5\%) under ImageNet linear evaluation protocol using small-batch size(\eg, 128), without requiring large-batch training on special hardware like TPU or inefficient across GPU operation (\eg, shuffling BN, synced BN). Our implementation and pre-trained models will be given on GitHub\footnote{https://github.com/zengarden/momentum2-teacher}.

11 pages, Tech report

References in corpus (6)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Large Batch Training of Convolutional Networks
• Prototypical Contrastive Learning of Unsupervised Representations
• Exploring Simple Siamese Representation Learning

Cited by in corpus (4)

• Rethinking "Batch" in BatchNorm
• Bag of Tricks and A Strong baseline for Image Copy Detection
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Voxel-wise Cross-Volume Representation Learning for 3D Neuron Reconstruction