73 citations · 146 across the 23 of their papers we have counts for
24 papers
Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn, Xiang Cheng, Minhak Song +3
Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of trai…
A Unified Approach to Controlling Implicit Regularization via Mirror Descent
Haoyuan Sun, Khashayar Gatmiry, Kwangjun Ahn +1
Inspired by the remarkable success of large neural networks, there has been significant interest in understanding the generalization performance of over-parameterized models. Subst…
On the Sample Complexity of Imitation Learning for Smoothed Model Predictive Control
Daniel Pfrommer, Swati Padmanabhan, Kwangjun Ahn +4
Recent work in imitation learning has shown that having an expert controller that is both suitably smooth and stable enables stronger guarantees on the performance of the learned c…
Transformers learn to implement preconditioned gradient descent for in-context learning
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand +1
Several recent works demonstrate that transformers can implement algorithms like gradient descent. By a careful construction of weights, these works show that multiple layers of tr…
How to escape sharp minima with random perturbations
Kwangjun Ahn, Ali Jadbabaie, Suvrit Sra
Modern machine learning applications have witnessed the remarkable success of optimization algorithms that are designed to find flat minima. Motivated by this design choice, we und…
The Crucial Role of Normalization in Sharpness-Aware Minimization
Yan Dai, Kwangjun Ahn, Suvrit Sra
Sharpness-Aware Minimization (SAM) is a recently proposed gradient-based optimizer (Foret et al., ICLR 2021) that greatly improves the prediction performance of deep neural network…