Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Abstract

Normalization techniques such as Batch Normalization have been applied successfully for training deep neural networks. Yet, despite its apparent empirical benefits, the reasons behind the success of Batch Normalization are mostly hypothetical. We here aim to provide a more thorough theoretical understanding from a classical optimization perspective. Our main contribution towards this goal is the identification of various problem instances in the realm of machine learning where % -- under certain assumptions-- Batch Normalization can provably accelerate optimization. We argue that this acceleration is due to the fact that Batch Normalization splits the optimization task into optimizing length and direction of the parameters separately. This allows gradient-based methods to leverage a favourable global structure in the loss landscape that we prove to exist in Learning Halfspace problems and neural network training with Gaussian inputs. We thereby turn Batch Normalization from an effective practical heuristic into a provably converging algorithm for these settings. Furthermore, we substantiate our analysis with empirical evidence that suggests the validity of our theoretical results in a broader context.

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• How Does Batch Normalization Help Optimization?
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• Accelerated Methods for Non-Convex Optimization
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima
• Learning Halfspaces and Neural Networks with Random Initialization
• Convergence theory for preconditioned eigenvalue solvers in a nutshell

Cited by in corpus (18)

• How Does Batch Normalization Help Optimization?
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• A Mean Field Theory of Batch Normalization
• An Exponential Learning Rate Schedule for Deep Learning
• Unsupervised Out-of-Distribution Detection with Batch Normalization
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Batch Normalization Orthogonalizes Representations in Deep Random Networks
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Information Geometry of Orthogonal Initializations and Training
• Depth Reconstruction of Translucent Objects from a Single Time-of-Flight Camera using Deep Residual Networks
• Dimensional Reweighting Graph Convolutional Networks
• Deep Curvature Suite
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Accelerating Training of Deep Neural Networks with a Standardization Loss
• An Internal Covariate Shift Bounding Algorithm for Deep Neural Networks by Unitizing Layers' Outputs
• An Investigation into the Stochasticity of Batch Whitening