Four Things Everyone Should Know to Improve Batch Normalization

Abstract

A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters gamma and beta; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.

arXiv commentICLR 2020, 18 pages

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• L2 Regularization versus Batch and Weight Normalization
• Differentiable Learning-to-Normalize via Switchable Normalization
• Measuring the Effects of Data Parallelism on Neural Network Training
• Norm matters: efficient and accurate normalization schemes in deep networks
• Decorrelated Batch Normalization
• Regularizing by the Variance of the Activations' Sample-Variances
• Mode Normalization

Cited by in corpus (12)

• High-Performance Large-Scale Image Recognition Without Normalization
• LTD: Low Temperature Distillation for Gradient Masking-free Adversarial Training
• Batch Group Normalization
• Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization
• Evaluation of Data Augmentation and Loss Functions in Semantic Image Segmentation for Drilling Tool Wear Detection
• Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Batch Normalization Preconditioning for Neural Network Training
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations
• Post-Training BatchNorm Recalibration
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization