44 citations · 82 across the 5 of their papers we have counts for
14 papers · 1 filter
Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn, Xiang Cheng, Minhak Song +3
Transformer training is notoriously difficult, requiring a careful design of optimizers and use of various heuristics. We make progress towards understanding the subtleties of trai…
Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory
Minhak Song, Chulhee Yun
Cohen et al. (2021) empirically study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent (GD) trajectory and observe t…
PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning
Hojoon Lee, Hanseul Cho, Hyunseung Kim +5
In Reinforcement Learning (RL), enhancing sample efficiency is crucial, particularly in scenarios when data acquisition is costly and risky. In principle, off-policy RL algorithms…
Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima
Dongkuk Si, Chulhee Yun
Sharpness-Aware Minimization (SAM) is an optimizer that takes a descent step based on the gradient at a perturbation $y_t = x_t + ρ\frac{\nabla f(x_t)}{\lVert \nabla f(x_t) \rVert}…
Can Single-Shuffle SGD be Better than Reshuffling SGD and GD?
Chulhee Yun, Suvrit Sra, Ali Jadbabaie
We propose matrix norm inequalities that extend the Recht-Ré (2012) conjecture on a noncommutative AM-GM inequality by supplementing it with another inequality that accounts for si…
Provable Memorization via Deep Neural Networks using Sub-linear Parameters
Sejun Park, Jaeho Lee, Chulhee Yun +1
It is known that parameters are sufficient for neural networks to memorize arbitrary input-label pairs. By exploiting depth, we show that parameters suffice…