An Investigation into Neural Net Optimization via Hessian Eigenvalue Density

Behrooz Ghorbani, Shankar Krishnan, Ying Xiao

To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Usi…