SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation

Robert M. Gower, Othmane Sebbouh, Nicolas Loizou

Stochastic Gradient Descent (SGD) is being used routinely for optimizing non-convex functions. Yet, the standard convergence theory for SGD in the smooth non-convex setting gives a…