Universal characteristics of deep neural network loss surfaces from random matrix theory
arXiv:2205.08601 · doi:10.1088/1751-8121/aca7f5
Abstract
This paper considers several aspects of random matrix universality in deep neural networks. Motivated by recent experimental work, we use universal properties of random matrices related to local statistics to derive practical implications for deep neural networks based on a realistic model of their Hessians. In particular we derive universal aspects of outliers in the spectra of deep neural networks and demonstrate the important role of random matrix local laws in popular pre-conditioning gradient descent algorithms. We also present insights into deep neural network loss surfaces from quite general arguments based on tools from statistical physics and random matrix theory.
arXiv comment42 pages
References in corpus (23)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- The Loss Surfaces of Multilayer Networks
- Visualizing the Loss Landscape of Neural Nets
- GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
- Random Matrices with Slow Correlation Decay
- The Emergence of Spectral Universality in Deep Networks
- Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions
- Local Kesten--McKay law for random regular graphs
- Comparing Dynamics: Deep Neural Networks versus Glassy Systems
- Explorations on high dimensional landscapes
- Characteristic polynomials for 1D random band matrices from the localization side
- Eigenvectors distribution and quantum unique ergodicity for deformed Wigner matrices
- Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
- Spectrum of deformed random matrices and free probability
- Fluctuations in local quantum unique ergodicity for generalized Wigner matrices
- The Loss Surfaces of Neural Networks with General Activation Functions
- On Random Matrices Arising in Deep Neural Networks. Gaussian Case
- Random matrix analysis of deep neural network weight matrices
- A spin-glass model for the loss surfaces of generative adversarial networks
- More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize
- Characteristic polynomials for random band matrices near the threshold
- Iterative Averaging in the Quest for Best Test Error
- On Random Matrices Arising in Deep Neural Networks: General I.I.D. Case