Hessian Eigenspectra of More Realistic Nonlinear Models
arXiv:2103.01519
Abstract
Given an optimization problem, the Hessian matrix and its eigenspectrum can be used in many ways, ranging from designing more efficient second-order algorithms to performing model analysis and regression diagnostics. When nonlinear models and non-convex problems are considered, strong simplifying assumptions are often made to make Hessian spectral analysis more tractable. This leads to the question of how relevant the conclusions of such analyses are for more realistic nonlinear models. In this paper, we exploit deterministic equivalent techniques from random matrix theory to make a \emph{precise} characterization of the Hessian eigenspectra for a broad family of nonlinear models, including models that generalize the classical generalized linear models, without relying on strong simplifying assumptions used previously. We show that, depending on the data properties, the nonlinear response model, and the loss function, the Hessian can have \emph{qualitatively} different spectral behaviors: of bounded or unbounded support, with single- or multi-bulk, and with isolated eigenvalues on the left- or right-hand side of the bulk. By focusing on such a simple but nontrivial nonlinear model, our analysis takes a step forward to unveil the theoretical origin of many visually striking features observed in more complex machine learning models.
arXiv commentIdentical to v1, except for the inclusion of some additional references
References in corpus (24)
- The Loss Surfaces of Multilayer Networks
- Statistical physics of inference: Thresholds and algorithms
- Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview
- Sparse principal component analysis and iterative thresholding
- Sparse Principal Components Analysis
- Limit theorems for sample eigenvalues in a generalized spiked population model
- Edge Universality of Beta Ensembles
- The statistics of critical points of Gaussian fields on large-dimensional spaces
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
- Spectral Moments of Correlated Wishart Matrices
- The Heavy-Tail Phenomenon in SGD
- Multiplicative noise and heavy tails in stochastic optimization
- PyHessian: Neural Networks Through the Lens of the Hessian
- Hessian spectrum at the global minimum of high-dimensional random landscapes
- Emergent properties of the local geometry of neural loss landscapes
- Hessian eigenvalue distribution in a random Gaussian landscape
- Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
- Average-case Acceleration Through Spectral Density Estimation
- Beyond Random Matrix Theory for Deep Networks
- Sparse Quantized Spectral Clustering