Theoretical Analysis of Auto Rate-Tuning by Batch Normalization

Abstract

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, $0.3$), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of $T^{-1/2}$ in $T$ iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate $T^{-1/4}$ is also shown for stochastic gradient descent.

arXiv comment22 pages

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• WNGrad: Learn the Learning Rate in Gradient Descent
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

Cited by in corpus (30)

• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
• State estimation with limited sensors -- A deep learning based approach
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Angle-based Search Space Shrinking for Neural Architecture Search
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Few-shot Neural Architecture Search
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Positional Normalization
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Batch Normalization Preconditioning for Neural Network Training
• Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
• A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
• Implicit Regularization and Convergence for Weight Normalization
• Stochastic Optimization with Laggard Data Pipelines
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• New Interpretations of Normalization Methods in Deep Learning
• An Investigation into the Stochasticity of Batch Whitening
• Data-efficient Alignment of Multimodal Sequences by Aligning Gradient Updates and Internal Feature Distributions
• Characterization of Excess Risk for Locally Strongly Convex Population Risk