Eigenvalues of the Hessian in Deep Learning: The Origin of Symmetry and Its Breaking

Yossi Arjevani

Hessian spectra at trained models in deep learning exhibit a persistent pattern: eigenvalues organize into distinct clusters, including a large bulk near zero and a few isolated ou…