Sharpness-Aware Minimization and the Edge of Stability

Philip M. Long, Peter L. Bartlett

Recent experiments have shown that, often, when training a neural network with gradient descent (GD) with a step size $\hat{I}\cdot$, the operator norm of the Hessian of the loss grows until…