Negative eigenvalues of the Hessian in deep neural networks

Abstract

The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.

References in corpus (5)

• Understanding deep learning requires rethinking generalization
• The Loss Surfaces of Multilayer Networks
• Gradient Descent Happens in a Tiny Subspace
• Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems
• Stochastic Cubic Regularization for Fast Nonconvex Optimization

Cited by in corpus (3)

• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Revisiting the Fragility of Influence Functions