209 citations · 415 across the 30 of their papers we have counts for
5 papers · 1 filter
Variance-Reduced Methods for Machine Learning
Robert M. Gower, Mark Schmidt, Francis Bach +1
Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago. The last 8 years hav…
SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation
Robert M. Gower, Othmane Sebbouh, Nicolas Loizou
Stochastic Gradient Descent (SGD) is being used routinely for optimizing non-convex functions. Yet, the standard convergence theory for SGD in the smooth non-convex setting gives a…
Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball
Othmane Sebbouh, Robert M. Gower, Aaron Defazio
We study stochastic gradient descent (SGD) and the stochastic heavy ball method (SHB, otherwise known as the momentum method) for the general stochastic approximation problem. For…
The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimization
Aaron Defazio, Robert M. Gower
The convergence rates for convex and non-convex optimization methods depend on the choice of a host of constants, including step sizes, Lyapunov function constants and momentum con…
Fast Linear Convergence of Randomized BFGS
Dmitry Kovalev, Robert M. Gower, Peter Richtárik +1
Since the late 1950's when quasi-Newton methods first appeared, they have become one of the most widely used and efficient algorithmic paradigms for unconstrained optimization. Des…