On Learning Rates and Schrödinger Operators

Abstract

The learning rate is perhaps the single most important parameter in the training of neural networks and, more broadly, in stochastic (nonconvex) optimization. Accordingly, there are numerous effective, but poorly understood, techniques for tuning the learning rate, including learning rate decay, which starts with a large initial learning rate that is gradually decreased. In this paper, we present a general theoretical analysis of the effect of the learning rate in stochastic gradient descent (SGD). Our analysis is based on the use of a learning-rate-dependent stochastic differential equation (lr-dependent SDE) that serves as a surrogate for SGD. For a broad class of objective functions, we establish a linear rate of convergence for this continuous-time formulation of SGD, highlighting the fundamental importance of the learning rate in SGD, and contrasting to gradient descent and stochastic gradient Langevin dynamics. Moreover, we obtain an explicit expression for the optimal linear rate by analyzing the spectrum of the Witten-Laplacian, a special case of the Schrödinger operator associated with the lr-dependent SDE. Strikingly, this expression clearly reveals the dependence of the linear convergence rate on the learning rate -- the linear rate decreases rapidly to zero as the learning rate tends to zero for a broad class of nonconvex functions, whereas it stays constant for strongly convex functions. Based on this sharp distinction between nonconvex and convex problems, we provide a mathematical interpretation of the benefits of using learning rate decay for nonconvex optimization.

49 pages, 21 figures

References in corpus (12)

• ADADELTA: An Adaptive Learning Rate Method
• SGDR: Stochastic Gradient Descent with Warm Restarts
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Three Factors Influencing Minima in SGD
• How Does Learning Rate Decay Help Modern Neural Networks?
• Optimization for deep learning: theory and algorithms
• Understanding the Acceleration Phenomenon via High-Resolution Differential Equations
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• An Exponential Learning Rate Schedule for Deep Learning
• Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chains
• Dynamics of Langevin Simulation
• Robust Learning Rate Selection for Stochastic Optimization via Splitting Diagnostic

Cited by in corpus (8)

• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate
• On stochastic mirror descent with interacting particles: convergence properties and variance reduction
• On the Hyperparameters in Stochastic Gradient Descent with Momentum
• Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• Revisiting the Effects of Stochasticity for Hamiltonian Samplers