Analytic Insights into Structure and Rank of Neural Network Hessian Maps

Abstract

The Hessian of a neural network captures parameter interactions through second-order derivatives of the loss. It is a fundamental object of study, closely tied to various problems in deep learning, including model design, optimization, and generalization. Most prior work has been empirical, typically focusing on low-rank approximations and heuristics that are blind to the network structure. In contrast, we develop theoretical tools to analyze the range of the Hessian map, providing us with a precise understanding of its rank deficiency as well as the structural reasons behind it. This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks, allowing for an elegant interpretation in terms of rank deficiency. Moreover, we demonstrate that our bounds remain faithful as an estimate of the numerical Hessian rank, for a larger class of models such as rectified and hyperbolic tangent networks. Further, we also investigate the implications of model architecture (e.g.~width, depth, bias) on the rank deficiency. Overall, our work provides novel insights into the source and extent of redundancy in overparameterized networks.

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Theano: new features and speed improvements
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Shaping the learning landscape in neural networks around wide flat minima
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians