Towards Understanding Regularization in Batch Normalization

Abstract

Batch Normalization (BN) improves both convergence and generalization in training neural networks. This work understands these phenomena theoretically. We analyze BN by using a basic block of neural networks, consisting of a kernel layer, a BN layer, and a nonlinear activation function. This basic network helps us understand the impacts of BN in three aspects. First, by viewing BN as an implicit regularizer, BN can be decomposed into population normalization (PN) and gamma decay as an explicit regularization. Second, learning dynamics of BN and the regularization show that training converged with large maximum and effective learning rate. Third, generalization of BN is explored by using statistical mechanics. Experiments demonstrate that BN in convolutional neural networks share the same traits of regularization as the above analyses.

arXiv commentInternational Conference on Learning Representations (ICLR)

Cited by in corpus (23)

• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• Quantifying Generalization in Reinforcement Learning
• Differentiable Learning-to-Normalize via Switchable Normalization
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Batch Group Normalization
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Positional Normalization
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Batch Normalization Preconditioning for Neural Network Training
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Implicit Regularization and Convergence for Weight Normalization
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Multirate Training of Neural Networks
• End-to-End Deep Residual Learning with Dilated Convolutions for Myocardial Infarction Detection and Localization
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Group Whitening: Balancing Learning Efficiency and Representational Capacity