Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Abstract

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show that this key benefit arises because, at initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor on the order of the square root of the network depth. This ensures that, early in training, the function computed by normalized residual blocks in deep networks is close to the identity function (on average). We use this insight to develop a simple initialization scheme that can train deep residual networks without normalization. We also provide a detailed empirical study of residual networks, which clarifies that, although batch normalized networks can be trained with larger learning rates, this effect is only beneficial in specific compute regimes, and has minimal benefits when the batch size is small.

arXiv commentCamera-ready version of NeurIPS 2020

References in corpus (21)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Don't Decay the Learning Rate, Increase the Batch Size
• Self-training with Noisy Student improves ImageNet classification
• Three Factors Influencing Minima in SGD
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• Fixup Initialization: Residual Learning Without Normalization
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• A Mean Field Theory of Batch Normalization
• Towards Understanding Regularization in Batch Normalization
• An Exponential Learning Rate Schedule for Deep Learning
• The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent
• Deep Residual Networks and Weight Initialization
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts

Cited by in corpus (11)

• ReZero is All You Need: Fast Convergence at Large Depth
• Evolving Normalization-Activation Layers
• Continuous-in-Depth Neural Networks
• Batch Group Normalization
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization
• Speeding up Deep Model Training by Sharing Weights and Then Unsharing
• The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• Implicit regularization of deep residual networks towards neural ODEs