A Mean Field Theory of Batch Normalization

Abstract

We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and gradient backpropagation in wide batch-normalized networks at initialization. Our theory shows that gradient signals grow exponentially in depth and that these exploding gradients cannot be eliminated by tuning the initial weight variances or by adjusting the nonlinear activation function. Indeed, batch normalization itself is the cause of gradient explosion. As a result, vanilla batch-normalized networks without skip connections are not trainable at large depths for common initialization schemes, a prediction that we verify with a variety of empirical simulations. While gradient explosion cannot be eliminated, it can be reduced by tuning the network close to the linear regime, which improves the trainability of deep batch-normalized networks without residual connections. Finally, we investigate the learning dynamics of batch-normalized networks and observe that after a single step of optimization the networks achieve a relatively stable equilibrium in which gradients have dramatically smaller dynamic range. Our theory leverages Laplace, Fourier, and Gegenbauer transforms and we derive new identities that may be of independent interest.

To appear in ICLR 2019

References in corpus (13)

• Deep Neural Networks as Gaussian Processes
• Exponential expressivity in deep neural networks through transient chaos
• How Does Batch Normalization Help Optimization?
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep Convolutional Networks as shallow Gaussian Processes
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• How to Start Training: The Effect of Initialization and Architecture
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks

Cited by in corpus (34)

• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Review: Deep Learning in Electron Microscopy
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Towards Efficient Training for Neural Network Quantization
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Feature Learning in Infinite-Width Neural Networks
• Tensor Programs III: Neural Matrix Laws
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Robust Pruning at Initialization
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Information Geometry of Orthogonal Initializations and Training
• Batch Normalization Preconditioning for Neural Network Training
• Initialization of ReLUs for Dynamical Isometry
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Picking Up Quantization Steps for Compressed Image Classification
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Improving Unsupervised Domain Adaptation by Reducing Bi-level Feature Redundancy
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theory
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• How Does BN Increase Collapsed Neural Network Filters?
• New Interpretations of Normalization Methods in Deep Learning