Distributed Gradient Methods for Nonconvex Optimization: Local and Global Convergence Guarantees
arXiv:2003.10309
Abstract
The article discusses distributed gradient-descent algorithms for computing local and global minima in nonconvex optimization. For local optimization, we focus on distributed stochastic gradient descent (D-SGD)--a simple network-based variant of classical SGD. We discuss local minima convergence guarantees and explore the simple but critical role of the stable-manifold theorem in analyzing saddle-point avoidance. For global optimization, we discuss annealing-based methods in which slowly decaying noise is added to D-SGD. Conditions are discussed under which convergence to global minima is guaranteed. Numerical examples illustrate the key concepts in the paper.
References in corpus (7)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- A Variational Perspective on Accelerated Methods in Optimization
- Matrix Completion has No Spurious Local Minimum
- When Are Nonconvex Problems Not Scary?
- Distributed Stochastic Gradient Descent: Nonconvexity, Nonsmoothness, and Convergence to Local Minima
- On Stationary-Point Hitting Time and Ergodicity of Stochastic Gradient Langevin Dynamics