An Improved Analysis of Stochastic Gradient Descent with Momentum

Abstract

SGD with momentum (SGDM) has been widely applied in many machine learning tasks, and it is often applied with dynamic stepsizes and momentum weights tuned in a stagewise manner. Despite of its empirical advantage over SGD, the role of momentum is still unclear in general since previous analyses on SGDM either provide worse convergence bounds than those of SGD, or assume Lipschitz or quadratic objectives, which fail to hold in practice. Furthermore, the role of dynamic parameters has not been addressed. In this work, we show that SGDM converges as fast as SGD for smooth objectives under both strongly convex and nonconvex settings. We also establish \textit{the first} convergence guarantee for the multistage setting, and show that the multistage strategy is beneficial for SGDM compared to using fixed parameters. Finally, we verify these theoretical claims by numerical experiments.

References in corpus (5)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Optimization for deep learning: theory and algorithms
• Understanding the Role of Momentum in Stochastic Gradient Methods
• Linearly convergent stochastic heavy ball method for minimizing generalization error

Cited by in corpus (5)

• Learning from History for Byzantine Robust Optimization
• A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques
• Uniform-in-time propagation of chaos for kinetic mean field Langevin dynamics
• An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models
• Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization