A Diffusion Approximation Theory of Momentum SGD in Nonconvex Optimization

Abstract

Momentum Stochastic Gradient Descent (MSGD) algorithm has been widely applied to many nonconvex optimization problems in machine learning, e.g., training deep neural networks, variational Bayesian inference, and etc. Despite its empirical success, there is still a lack of theoretical understanding of convergence properties of MSGD. To fill this gap, we propose to analyze the algorithmic behavior of MSGD by diffusion approximations for nonconvex optimization problems with strict saddle points and isolated local optima. Our study shows that the momentum helps escape from saddle points, but hurts the convergence within the neighborhood of optima (if without the step size annealing or momentum annealing). Our theoretical discovery partially corroborates the empirical success of MSGD in training deep neural networks.

arXiv commentarXiv admin note: text overlap with arXiv:1806.01660

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• The Loss Surfaces of Multilayer Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
• Towards Understanding the Importance of Noise in Training Neural Networks
• Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms
• Dimensionality Reduction for Stationary Time Series via Stochastic Nonconvex Optimization
• Acceleration and Averaging in Stochastic Mirror Descent Dynamics

Cited by in corpus (3)

• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Models