Can Stability be Detrimental? Better Generalization through Gradient Descent Instabilities

Lawrence Wang, Stephen J. Roberts

Traditional analyses of gradient descent optimization show that, when the largest eigenvalue of the loss Hessian - often referred to as the sharpness - is below a critical learning…