77 citations · 318 across the 57 of their papers we have counts for
4 papers · 1 filter
High-dimensional Limit of SGD for Diagonal Linear Networks
Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1
Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet…
Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy +1
We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction.…
A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
Damek Davis, Dmitriy Drusvyatskiy
Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically…
When do spectral gradient updates help in deep learning?
Damek Davis, Dmitriy Drusvyatskiy
Spectral gradient methods, such as the recently popularized Muon optimizer, are a promising alternative to standard Euclidean gradient descent for training deep neural networks and…