36 citations · 63 across the 7 of their papers we have counts for
6 papers · 1 filter
Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models
Alex Damian, Eshaan Nichani, Rong Ge +1
We focus on the task of learning a single index model with respect to the isotropic Gaussian distribution in dimensions. Prior work has shown that the samp…
Fine-Tuning Language Models with Just Forward Passes
Sadhika Malladi, Tianyu Gao, Eshaan Nichani +4
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-…
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Eshaan Nichani, Alex Damian, Jason D. Lee
One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient featur…
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Alex Damian, Eshaan Nichani, Jason D. Lee
Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness , is bounded by , training is "stable" and the…
Neural Networks can Learn Representations with Gradient Descent
Alex Damian, Jason D. Lee, Mahdi Soltanolkotabi
Significant theoretical work has established that in specific regimes, neural networks trained by gradient descent behave like kernel methods. However, in practice, it is known tha…
Label Noise SGD Provably Prefers Flat Global Minimizers
Alex Damian, Tengyu Ma, Jason D. Lee
In overparametrized models, the noise in stochastic gradient descent (SGD) implicitly regularizes the optimization trajectory and determines which local minimum SGD converges to. M…