A Loss Curvature Perspective on Training Instability in Deep Learning

Justin Gilmer, Behrooz Ghorbani, Ankush Garg +6

In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics.…