927 citations · 2.1k across the 31 of their papers we have counts for
Showing 2021Show all
2 papers · 1 filter
cs.LG2021
Stochastic Polyak Stepsize with a Moving Target
Robert M. Gower, Aaron Defazio, Michael Rabbat
We propose a new stochastic gradient method called MOTAPS (Moving Targetted Polyak Stepsize) that uses recorded past loss values to compute adaptive stepsizes. MOTAPS can be seen a…
cs.LG2021★ 40 cited
Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization
Aaron Defazio, Samy Jelassi
We introduce MADGRAD, a novel optimization method in the family of AdaGrad adaptive gradient methods. MADGRAD shows excellent performance on deep learning optimization problems fro…