Training Instabilities Induce Flatness Bias in Gradient Descent

Lawrence Wang, Stephen J. Roberts

Classical analyses of gradient descent (GD) define a stability threshold based on the largest eigenvalue of the loss Hessian, often termed sharpness. When the learning rate lies be…