209 citations · 415 across the 30 of their papers we have counts for
9 papers · 1 filter
Function Value Learning: Adaptive Learning Rates Based on the Polyak Stepsize and Function Splitting in ERM
Guillaume Garrigos, Robert M. Gower, Fabian Schaipp
Here we develop variants of SGD (stochastic gradient descent) with an adaptive step size that make use of the sampled loss values. In particular, we focus on solving a finite sum-o…
Variational Inference with Gaussian Score Matching
Chirag Modi, Charles Margossian, Yuling Yao +3
Variational inference (VI) is a method to approximate the computationally intractable posterior distributions that arise in Bayesian statistics. Typically, VI fits a simple paramet…
Provable convergence guarantees for black-box variational inference
Justin Domke, Guillaume Garrigos, Robert Gower
Black-box variational inference is widely used in situations where there is no proof that its stochastic optimization succeeds. We suggest this is due to a theoretical gap in exist…
A Model-Based Method for Minimizing CVaR and Beyond
Si Yi Meng, Robert M. Gower
We develop a variant of the stochastic prox-linear method for minimizing the Conditional Value-at-Risk (CVaR) objective. CVaR is a risk measure focused on minimizing worst-case per…
Improving Convergence and Generalization Using Parameter Symmetries
Bo Zhao, Robert M. Gower, Robin Walters +1
In many neural networks, different values of the parameters may result in the same loss value. Parameter space symmetries are loss-invariant transformations that change the model p…
MoMo: Momentum Models for Adaptive Learning Rates
Fabian Schaipp, Ruben Ohana, Michael Eickenberg +2
Training a modern machine learning architecture on a new task requires extensive learning-rate tuning, which comes at a high computational cost. Here we develop new Polyak-type ada…
A Bregman-Kaczmarz method for nonlinear systems of equations
Robert Gower, Dirk A. Lorenz, Maximilian Winkler
We propose a new randomized method for solving systems of nonlinear equations, which can find sparse solutions or solutions under certain simple constraints. The scheme only takes…
Handbook of Convergence Theorems for (Stochastic) Gradient Methods
Guillaume Garrigos, Robert M. Gower
This is a handbook of simple proofs of the convergence of gradient and stochastic gradient descent type methods. We consider functions that are Lipschitz, smooth, convex, strongly…
A Stochastic Proximal Polyak Step Size
Fabian Schaipp, Robert M. Gower, Michael Ulbrich
Recently, the stochastic Polyak step size (SPS) has emerged as a competitive adaptive step size scheme for stochastic gradient descent. Here we develop ProxSPS, a proximal variant…