Appearance of Random Matrix Theory in Deep Learning

Abstract

We investigate the local spectral statistics of the loss surface Hessians of artificial neural networks, where we discover excellent agreement with Gaussian Orthogonal Ensemble statistics across several network architectures and datasets. These results shed new light on the applicability of Random Matrix Theory to modelling neural networks and suggest a previously unrecognised role for it in the study of loss surfaces in deep learning. Inspired by these observations, we propose a novel model for the true loss surfaces of neural networks, consistent with our observations, which allows for Hessian spectral densities with rank degeneracy and outliers, extensively observed in practice, and predicts a growing independence of loss gradients as a function of distance in weight-space. We further investigate the importance of the true loss surface in neural networks and find, in contrast to previous work, that the exponential hardness of locating the global minimum has practical consequences for achieving state of the art performance.

arXiv comment33 pages, 14 figures

References in corpus (32)

• Random-Matrix Theory of Quantum Transport
• Machine learning and the physical sciences
• Random Matrix Theories in Quantum Physics: Common Concepts
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The distribution of the ratio of consecutive level spacings in random matrix ensembles
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Statistical physics of inference: Thresholds and algorithms
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Random Matrices and Chaos in Nuclear Physics
• New insights and perspectives on the natural gradient method
• The Principles of Deep Learning Theory
• Cleaning large correlation matrices: tools from random matrix theory
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Topology trivialization and large deviations for the minimum in the simplest random optimization
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Explorations on high dimensional landscapes
• Mean-field inference methods for neural networks
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• The Spectral Backbone of Excitation Transport in Ultra-Cold Rydberg Gases
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Hessian spectrum at the global minimum of high-dimensional random landscapes
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• A Precise Performance Analysis of Learning with Random Features
• Effect of Unfolding on the Spectral Statistics of Adjacency Matrices of Complex Networks
• The Loss Surfaces of Neural Networks with General Activation Functions
• On the energy landscape of deep networks
• A spin-glass model for the loss surfaces of generative adversarial networks
• Beyond Random Matrix Theory for Deep Networks