Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate

Abstract

Recent works (e.g., (Li and Arora, 2020)) suggest that the use of popular normalization schemes (including Batch Normalization) in today's deep learning can move it far from a traditional optimization viewpoint, e.g., use of exponentially increasing learning rates. The current paper highlights other ways in which behavior of normalized nets departs from traditional viewpoints, and then initiates a formal framework for studying their mathematics via suitable adaptation of the conventional framework namely, modeling SGD-induced training trajectory via a suitable stochastic differential equation (SDE) with a noise term that captures gradient noise. This yields: (a) A new ' intrinsic learning rate' parameter that is the product of the normal learning rate and weight decay factor. Analysis of the SDE shows how the effective speed of learning varies and equilibrates over time under the control of intrinsic LR. (b) A challenge -- via theory and experiments -- to popular belief that good generalization requires large learning rates at the start of training. (c) New experiments, backed by mathematical intuition, suggesting the number of steps to equilibrium (in function space) scales as the inverse of the intrinsic learning rate, as opposed to the exponential time convergence bound implied by SDE analysis. We name it the Fast Equilibrium Conjecture and suggest it holds the key to why Batch Normalization is effective.

25 pages, 12 figures. Accepted By 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

Topics & keywords

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• L2 Regularization versus Batch and Weight Normalization
• Essentially No Barriers in Neural Network Energy Landscape
• Sharp Minima Can Generalize For Deep Nets
• Norm matters: efficient and accurate normalization schemes in deep networks
• On Learning Rates and Schrödinger Operators
• Stochastic Gradient and Langevin Processes

Cited by in corpus (8)

• A Practical Survey on Faster and Lighter Transformers
• On the training dynamics of deep networks with $L_2$ regularization
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay
• Acceleration via Fractal Learning Rate Schedules