36 citations · 63 across the 7 of their papers we have counts for
8 papers
Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models
Alex Damian, Eshaan Nichani, Rong Ge +1
We focus on the task of learning a single index model with respect to the isotropic Gaussian distribution in dimensions. Prior work has shown that the samp…
Fine-Tuning Language Models with Just Forward Passes
Sadhika Malladi, Tianyu Gao, Eshaan Nichani +4
Fine-tuning language models (LMs) has yielded success on diverse downstream tasks, but as LMs grow in size, backpropagation requires a prohibitively large amount of memory. Zeroth-…
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks
Eshaan Nichani, Alex Damian, Jason D. Lee
One of the central questions in the theory of deep learning is to understand how neural networks learn hierarchical features. The ability of deep networks to extract salient featur…
Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability
Alex Damian, Eshaan Nichani, Jason D. Lee
Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness , is bounded by , training is "stable" and the…
Neural Networks can Learn Representations with Gradient Descent
Alex Damian, Jason D. Lee, Mahdi Soltanolkotabi
Significant theoretical work has established that in specific regimes, neural networks trained by gradient descent behave like kernel methods. However, in practice, it is known tha…
Label Noise SGD Provably Prefers Flat Global Minimizers
Alex Damian, Tengyu Ma, Jason D. Lee
In overparametrized models, the noise in stochastic gradient descent (SGD) implicitly regularizes the optimization trajectory and determines which local minimum SGD converges to. M…
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models
Sachit Menon, Alexandru Damian, Shijia Hu +2
The primary aim of single-image super-resolution is to construct high-resolution (HR) images from corresponding low-resolution (LR) inputs. In previous approaches, which have gener…
New Techniques for Preserving Global Structure and Denoising with Low Information Loss in Single-Image Super-Resolution
Yijie Bei, Alex Damian, Shijia Hu +3
This work identifies and addresses two important technical challenges in single-image super-resolution: (1) how to upsample an image without magnifying noise and (2) how to preserv…