77 citations · 318 across the 57 of their papers we have counts for
75 papers
High-dimensional Limit of SGD for Diagonal Linear Networks
Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1
Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet…
Average Gradient Outer Product in kernel regression provably recovers the central subspace for multi-index models
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy +1
We study a prototypical situation when a learned predictor can discover useful low-dimensional structure in data, while using fewer samples than are needed for accurate prediction.…
A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
Damek Davis, Dmitriy Drusvyatskiy
Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically…
When do spectral gradient updates help in deep learning?
Damek Davis, Dmitriy Drusvyatskiy
Spectral gradient methods, such as the recently popularized Muon optimizer, are a promising alternative to standard Euclidean gradient descent for training deep neural networks and…
Iteratively reweighted kernel machines efficiently learn sparse functions
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy +1
The impressive practical performance of neural networks is often attributed to their ability to learn low-dimensional data representations and hierarchical structure directly from…
Spectral norm bound for the product of random Fourier-Walsh matrices
Libin Zhu, Damek Davis, Dmitriy Drusvyatskiy +1
We consider matrix products of the form , where are normalized random Fourier-Walsh matrices. We identify an interesting polyn…