36 citations · 176 across the 39 of their papers we have counts for
3 papers · 1 filter
To Each Optimizer a Norm, To Each Norm its Generalization
Sharan Vaswani, Reza Babanezhad, Jose Gallego-Posada +3
We study the implicit regularization of optimization methods for linear models interpolating the training data in the under-parametrized and over-parametrized regimes. Since it is…
Adaptive Gradient Methods Converge Faster with Over-Parameterization (but you should do a line-search)
Sharan Vaswani, Issam Laradji, Frederik Kunstner +3
Adaptive gradient methods are typically used for training over-parameterized models. To better understand their behaviour, we study a simplistic setting -- smooth, convex losses wi…
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
Nicolas Loizou, Sharan Vaswani, Issam Laradji +1
We propose a stochastic variant of the classical Polyak step-size (Polyak, 1987) commonly used in the subgradient method. Although computing the Polyak step-size requires knowledge…