77 citations · 318 across the 57 of their papers we have counts for
59 papers · 1 filter
High-dimensional Limit of SGD for Diagonal Linear Networks
Begoña García Malaxechebarría, Courtney Paquette, Maryam Fazel +1
Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet…
A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity
Damek Davis, Dmitriy Drusvyatskiy
Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically…
Invariant Kernels: Rank Stabilization and Generalization Across Dimensions
Mateo Díaz, Dmitriy Drusvyatskiy, Jack Kendrick +1
Symmetry arises often when learning from high dimensional data. For example, data sets consisting of point clouds, graphs, and unordered sets appear routinely in contemporary appli…
Gradient descent with adaptive stepsize converges (nearly) linearly under fourth-order growth
Damek Davis, Dmitriy Drusvyatskiy, Liwei Jiang
A prevalent belief among optimization specialists is that linear convergence of gradient descent is contingent on the function growing quadratically away from its minimizers. In th…
The slope robustly determines convex functions
Aris Daniilidis, Dmitriy Drusvyatskiy
We show that the deviation between the slopes of two convex functions controls the deviation between the functions themselves. This result reveals that the slope -- a one dimension…
Asymptotic normality and optimality in nonsmooth stochastic approximation
Damek Davis, Dmitriy Drusvyatskiy, Liwei Jiang
In their seminal work, Polyak and Juditsky showed that stochastic approximation algorithms for solving smooth equations enjoy a central limit theorem. Moreover, it has since been a…