Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
arXiv:2101.08482
Abstract
We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN), which improves the performance of existing student-teacher based self- and semi-supervised learning techniques. Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. This design reduces the intrinsic cross-sample dependency of BN and enhances the generalization of the teacher. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet. These improvements are consistent across methods, network architectures, training duration, and datasets, demonstrating the general effectiveness of this technique. The code is available at https://github.com/amazon-research/exponential-moving-average-normalization.
arXiv commentaccepted by CVPR21 as Oral presentation
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Instance Normalization: The Missing Ingredient for Fast Stylization
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Improved Baselines with Momentum Contrastive Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Unsupervised Data Augmentation for Consistency Training
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Prototypical Contrastive Learning of Unsupervised Representations
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
- Snapshot Ensembles: Train 1, get M for free
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization