Rethinking "Batch" in BatchNorm
arXiv:2105.07576
Abstract
BatchNorm is a critical building block in modern convolutional neural networks. Its unique property of operating on "batches" instead of individual samples introduces significantly different behaviors from most other operations in deep learning. As a result, it leads to many hidden caveats that can negatively impact model's performance in subtle ways. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of "batch" in BatchNorm. By presenting these caveats and their mitigations, we hope this review can help researchers use BatchNorm more effectively.
arXiv commentTech report
References in corpus (25)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Mixed Precision Training
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
- On First-Order Meta-Learning Algorithms
- DropBlock: A regularization method for convolutional networks
- Revisiting Batch Normalization For Practical Domain Adaptation
- High-Performance Large-Scale Image Recognition Without Normalization
- Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
- Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
- ELF OpenGo: An Analysis and Open Reimplementation of AlphaZero
- Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization
- On Feature Normalization and Data Augmentation
- Domain-Specific Batch Normalization for Unsupervised Domain Adaptation
- Towards an Adversarially Robust Normalization Approach
- Cross-Iteration Batch Normalization
- Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
- Split Batch Normalization: Improving Semi-Supervised Learning under Domain Shift
- Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
- Post-Training BatchNorm Recalibration
Cited by in corpus (5)
- LOANet: A Lightweight Network Using Object Attention for Extracting Buildings and Roads from UAV Aerial Remote Sensing Images
- Learning from Temporal Gradient for Semi-supervised Action Recognition
- Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
- Training BatchNorm Only in Neural Architecture Search and Beyond
- New Perspective on Progressive GANs Distillation for One-class Novelty Detection