Negative eigenvalues of the Hessian in deep neural networks
arXiv:1902.02366
Abstract
The loss function of deep networks is known to be non-convex but the precise nature of this nonconvexity is still an active area of research. In this work, we study the loss landscape of deep networks through the eigendecompositions of their Hessian matrix. In particular, we examine how important the negative eigenvalues are and the benefits one can observe in handling them appropriately.
References in corpus (7)
- Understanding deep learning requires rethinking generalization
- The Loss Surfaces of Multilayer Networks
- Gradient Descent Happens in a Tiny Subspace
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems
- Stochastic Cubic Regularization for Fast Nonconvex Optimization
- Estimating the Spectral Density of Large Implicit Matrices
Cited by in corpus (7)
- The Break-Even Point on Optimization Trajectories of Deep Neural Networks
- Hessian-based toolbox for reliable and interpretable machine learning in physics
- Visualizing high-dimensional loss landscapes with Hessian directions
- Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
- Revisiting the Fragility of Influence Functions
- DDPNOpt: Differential Dynamic Programming Neural Optimizer
- Low Rank Saddle Free Newton: A Scalable Method for Stochastic Nonconvex Optimization