209 citations · 415 across the 30 of their papers we have counts for
13 papers · 1 filter
Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM
Guillaume Garrigos, Robert M. Gower, Fabian Schaipp
Here we develop variants of SGD (stochastic gradient descent) with an adaptive step size that make use of the sampled loss values. In particular, we focus on solving a finite sum-o…
Provable convergence guarantees for black-box variational inference
Justin Domke, Guillaume Garrigos, Robert Gower
Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in exist…
Improving Convergence and Generalization Using Parameter Symmetries
Bo Zhao, Robert M. Gower, Robin Walters +1
In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model p…
MoMo: Momentum Models for Adaptive Learning Rates
Fabian Schaipp, Ruben Ohana, Michael Eickenberg +2
Training a modern machine learning architecture on a new task requires extensive learning-rate tuning, which comes at a high computational cost. Here we develop new Polyak-type ada…
Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies
Rui Yuan, Simon S. Du, Robert M. Gower +2
We consider infinite-horizon discounted Markov decision processes and study the convergence rates of the natural policy gradient (NPG) and the Q-NPG methods with the log-linear pol…
SP2: A Second Order Stochastic Polyak Method
Shuang Li, William J. Swartworth, Martin Takáč +2
Recently the "SP" (Stochastic Polyak step size) method has emerged as a competitive adaptive method for setting the step sizes of SGD. SP can be interpreted as a method specialized…
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
Robert M. Gower, Mathieu Blondel, Nidham Gazagnadou +1
Tuning the step size of stochastic gradient descent is tedious and error prone. This has motivated the development of methods that automatically adapt the step size using readily a…
A general sample complexity analysis of vanilla policy gradient
Rui Yuan, Robert M. Gower, Alessandro Lazaric
We adapt recent tools developed for the analysis of Stochastic Gradient Descent (SGD) in non-convex optimization to obtain convergence and sample complexity guarantees for the vani…
Stochastic Polyak Stepsize with a Moving Target
Robert M. Gower, Aaron Defazio, Michael Rabbat
We propose a new stochastic gradient method called MOTAPS (Moving Targetted Polyak Stepsize) that uses recorded past loss values to compute adaptive stepsizes. MOTAPS can be seen a…
Variance-Reduced Methods for Machine Learning
Robert M. Gower, Mark Schmidt, Francis Bach +1
Stochastic optimization lies at the heart of machine learning, and its cornerstone is stochastic gradient descent (SGD), a method introduced over 60 years ago. The last 8 years hav…
Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball
Othmane Sebbouh, Robert M. Gower, Aaron Defazio
We study stochastic gradient descent (SGD) and the stochastic heavy ball method (SHB, otherwise known as the momentum method) for the general stochastic approximation problem. For…
The Power of Factorial Powers: New Parameter settings for (Stochastic) Optimization
Aaron Defazio, Robert M. Gower
The convergence rates for convex and non-convex optimization methods depend on the choice of a host of constants, including step sizes, Lyapunov function constants and momentum con…
SGD: General Analysis and Improved Rates
Robert Mansel Gower, Nicolas Loizou, Xun Qian +3
We propose a general yet simple theorem describing the convergence of SGD under the arbitrary sampling paradigm. Our theorem describes the convergence of an infinite array of varia…