How Does Batch Normalization Help Optimization?

Abstract

Batch Normalization (BatchNorm) is a widely adopted technique that enables faster and more stable training of deep neural networks (DNNs). Despite its pervasiveness, the exact reasons for BatchNorm's effectiveness are still poorly understood. The popular belief is that this effectiveness stems from controlling the change of the layers' input distributions during training to reduce the so-called "internal covariate shift". In this work, we demonstrate that such distributional stability of layer inputs has little to do with the success of BatchNorm. Instead, we uncover a more fundamental impact of BatchNorm on the training process: it makes the optimization landscape significantly smoother. This smoothness induces a more predictive and stable behavior of the gradients, allowing for faster training.

arXiv commentIn NeurIPS'18

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Layer Normalization
• On the importance of single directions for generalization
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Identity Matters in Deep Learning
• An empirical analysis of the optimization of deep network loss surfaces
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Cited by in corpus (34)

• Fixup Initialization: Residual Learning Without Normalization
• Weak lensing cosmology with convolutional neural networks on noisy data
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• How to train your MAML
• Morphological classification of galaxies with deep learning: comparing 3-way and 4-way CNNs
• A Mean Field Theory of Batch Normalization
• Understanding the impact of entropy on policy optimization
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Deep Neural Networks for Marine Debris Detection in Sonar Images
• U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instrument
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Learning to learn via Self-Critique
• AnalogVNN: A fully modular framework for modeling and optimizing photonic neural networks
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Deep Learning with CNNs: A Compact Holistic Tutorial with Focus on Supervised Regression (Preprint)
• Information Geometry of Orthogonal Initializations and Training
• Truth or Backpropaganda? An Empirical Investigation of Deep Learning Theory
• Depth Reconstruction of Translucent Objects from a Single Time-of-Flight Camera using Deep Residual Networks
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• Automatic Segmentation of Aircraft Dents in Point Clouds
• Churn Reduction via Distillation
• Locally Adaptive Label Smoothing for Predictive Churn
• Second-Order Neural ODE Optimizer
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• Normalization Before Shaking Toward Learning Symmetrically Distributed Representation Without Margin in Speech Emotion Recognition
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Unsupervised Data Selection for Supervised Learning
• Learning Sampling Policy for Faster Derivative Free Optimization
• Sample Variance Decay in Randomly Initialized ReLU Networks
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Deep Representation with ReLU Neural Networks
• Efficient Modelling Across Time of Human Actions and Interactions