Understanding Batch Normalization

Abstract

Batch normalization (BN) is a technique to normalize activations in intermediate layers of deep neural networks. Its tendency to improve accuracy and speed up training have established BN as a favorite technique in deep learning. Yet, despite its enormous success, there remains little consensus on the exact reason and mechanism behind these improvements. In this paper we take a step towards a better understanding of BN, following an empirical approach. We conduct several experiments, and show that BN primarily enables training with larger learning rates, which is the cause for faster convergence and better generalization. For networks without BN we demonstrate how large gradient updates can result in diverging loss and activations growing uncontrollably with network depth, which limits possible learning rates. BN avoids this problem by constantly correcting activations to be zero-mean and of unit standard deviation, which enables larger gradient steps, yields faster convergence and may help bypass sharp local minima. We further show various ways in which gradients and activations of deep unnormalized networks are ill-behaved. We contrast our results against recent findings in random matrix theory, shedding new light on classical initialization schemes and their consequences.

References in corpus (18)

• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Understanding deep learning requires rethinking generalization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• The Loss Surfaces of Multilayer Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Self-Normalizing Neural Networks
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Revisiting Small Batch Training for Deep Neural Networks
• Three Factors Influencing Minima in SGD
• All you need is a good init
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Depth Creates No Bad Local Minima
• Recurrent Batch Normalization
• Global optimality conditions for deep neural networks
• Solving internal covariate shift in deep learning with linked neurons

Cited by in corpus (6)

• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction
• Deep Learning for direct Dark Matter search with nuclear emulsions
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• Learning Condensed and Aligned Features for Unsupervised Domain Adaptation Using Label Propagation
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control