On the Global Convergence of Continuous-Time Stochastic Heavy-Ball Method for Nonconvex Optimization
arXiv:1712.05733
Abstract
We study the convergence behavior of the stochastic heavy-ball method with a small stepsize. Under a change of time scale, we approximate the discrete method by a stochastic differential equation that models small random perturbations of a coupled system of nonlinear oscillators. We rigorously show that the perturbed system converges to a local minimum in a logarithmic time. This indicates that for the diffusion process that approximates the stochastic heavy-ball method, it takes (up to a logarithmic factor) only a linear time of the square root of the inverse stepsize to escape from all saddle points. This results may suggest a fast convergence of its discrete-time counterpart. Our theoretical results are validated by numerical experiments.
arXiv commentaccepted at IEEE International Conference on Big Data in 2019
References in corpus (5)
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- When Are Nonconvex Problems Not Scary?
- Escaping Saddles with Stochastic Gradients
- Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent
- On the fast convergence of random perturbations of the gradient flow