12 citations · 27 across the 9 of their papers we have counts for
4 papers · 1 filter
A Journey to the Edge of Stability
Jaerin Lee, Kyoung Mu Lee
It has recently been found that deep learning often occurs at the "edge of stability (EoS)," where the maximum Hessian eigenvalue of the model is stabilized at a value reciprocal t…
The Road Taken: The Role of Optimizers at the Edge of Stability
Jaerin Lee, Kyoung Mu Lee
The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the clas…
Greedy Alignment Principle for Optimizer Selection
Jaerin Lee, Kyoung Mu Lee
Recent works have shown that gradient-update alignment is a powerful signal for modulating optimizer updates, often leading to faster training. We promote this update-wise heuristi…
Grokfast: Accelerated Grokking by Amplifying Slow Gradients
Jaerin Lee, Bong Gyun Kang, Kihoon Kim +1
One puzzling artifact in machine learning dubbed grokking is where delayed generalization is achieved tenfolds of iterations after near perfect overfitting to the training data. Fo…