Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
arXiv:2010.04261
Abstract
Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. In this paper, we propose a decoupling conjecture that decomposes the layer-wise Hessians of a network as the Kronecker product of two smaller matrices. We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks. The decoupling conjecture has several other interesting implications - top eigenspaces for different models have surprisingly high overlap, and top eigenvectors form low rank matrices when they are reshaped into the same shape as the corresponding weight matrix. All of these can be verified empirically for deeper networks. Finally, we use the structure of layer-wise Hessian to get better explicit generalization bounds for neural networks.
arXiv comment72 pages, 31 figures. Main text: 10 pages, 7 figures. First two authors have equal contribution and are in alphabetical order
References in corpus (23)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Deep Neural Networks as Gaussian Processes
- Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Gradient Descent Happens in a Tiny Subspace
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Stiffness: A New Perspective on Generalization in Neural Networks
- On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
- PyHessian: Neural Networks Through the Lens of the Hessian
- EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
- The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
- Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
- Emergent properties of the local geometry of neural loss landscapes
- Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation
- A short note on the tail bound of Wishart distribution
- Pathological spectra of the Fisher information metric and its variants in deep neural networks
- Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
- Chain Rules for Hessian and Higher Derivatives Made Easy by Tensor Calculus