Towards Stabilizing Batch Statistics in Backward Propagation of Batch Normalization

Abstract

Batch Normalization (BN) is one of the most widely used techniques in Deep Learning field. But its performance can awfully degrade with insufficient batch size. This weakness limits the usage of BN on many computer vision tasks like detection or segmentation, where batch size is usually small due to the constraint of memory consumption. Therefore many modified normalization techniques have been proposed, which either fail to restore the performance of BN completely, or have to introduce additional nonlinear operations in inference procedure and increase huge consumption. In this paper, we reveal that there are two extra batch statistics involved in backward propagation of BN, on which has never been well discussed before. The extra batch statistics associated with gradients also can severely affect the training of deep neural network. Based on our analysis, we propose a novel normalization method, named Moving Average Batch Normalization (MABN). MABN can completely restore the performance of vanilla BN in small batch cases, without introducing any additional nonlinear operations in inference procedure. We prove the benefits of MABN by both theoretical analysis and experiments. Our experiments demonstrate the effectiveness of MABN in multiple computer vision tasks including ImageNet and COCO. The code has been released in https://github.com/megvii-model/MABN.

ICLR2020; https://github.com/megvii-model/MABN

References in corpus (2)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models

Cited by in corpus (13)

• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Rethinking "Batch" in BatchNorm
• PowerNorm: Rethinking Batch Normalization in Transformers
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Batch Group Normalization
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• A New Look at Ghost Normalization
• Unsupervised Model Drift Estimation with Batch Normalization Statistics for Dataset Shift Detection and Model Selection
• WeightAlign: Normalizing Activations by Weight Alignment