Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks

Abstract

Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. In this paper, we propose a decoupling conjecture that decomposes the layer-wise Hessians of a network as the Kronecker product of two smaller matrices. We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks. The decoupling conjecture has several other interesting implications - top eigenspaces for different models have surprisingly high overlap, and top eigenvectors form low rank matrices when they are reshaped into the same shape as the corresponding weight matrix. All of these can be verified empirically for deeper networks. Finally, we use the structure of layer-wise Hessian to get better explicit generalization bounds for neural networks.

72 pages, 31 figures. Main text: 10 pages, 7 figures. First two authors have equal contribution and are in alphabetical order

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Gradient Descent Happens in a Tiny Subspace
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• PyHessian: Neural Networks Through the Lens of the Hessian
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Emergent properties of the local geometry of neural loss landscapes
• Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation
• A short note on the tail bound of Wishart distribution
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Chain Rules for Hessian and Higher Derivatives Made Easy by Tensor Calculus

Cited by in corpus (5)

• Federated Optimization of Smooth Loss Functions
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Does the Data Induce Capacity Control in Deep Learning?
• Second-Order Neural ODE Optimizer
• Dynamic Game Theoretic Neural Optimizer