One-dimensional System Arising in Stochastic Gradient Descent
arXiv:1802.06760 · doi:10.1017/apr.2020.10
Abstract
We consider SDEs of the form , where behaves comparably to in a neighborhood of the origin, for . We show that there exists a threshold value for , depending on , such that when then , and for the rest of the permissible values . The previous results extend for discrete processes that satisfy . Here, are martingale differences that are a.s. bounded. This result shows that for a function , whose second derivative at degenerate saddle points is of polynomial order, it is always possible to escape saddle points via the iteration for a suitable choice of .
References in corpus (10)
- A survey of random processes with reinforcement
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Guaranteed Matrix Completion via Non-convex Factorization
- Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
- How to Escape Saddle Points Efficiently
- Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
- Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysis
- Gradient Descent Converges to Minimizers
- Efficient approaches for escaping higher order saddle points in non-convex optimization
- Escaping Saddles with Stochastic Gradients