Random matrix analysis of deep neural network weight matrices

Abstract

Neural networks have been used successfully in a variety of fields, which has led to a great deal of interest in developing a theoretical understanding of how they store the information needed to perform a particular task. We study the weight matrices of trained deep neural networks using methods from random matrix theory (RMT) and show that the statistics of most of the singular values follow universal RMT predictions. This suggests that they are random and do not contain system specific information, which we investigate further by comparing the statistics of eigenvector entries to the universal Porter-Thomas distribution. We find that for most eigenvectors the hypothesis of randomness cannot be rejected, and that only eigenvectors belonging to the largest singular values deviate from the RMT prediction, indicating that they may encode learned information. In addition, a comparison with RMT predictions also allows to distinguish networks trained in different learning regimes - from lazy to rich learning. We analyze the spectral distribution of the large singular values using the Hill estimator and find that the distribution cannot in general be characterized by a tail index, i.e. is not of power law type.

arXiv comment16 pages, 14 figures, updated version

References in corpus (29)

• Power-law distributions in empirical data
• Machine learning and the physical sciences
• Random Matrix Theories in Quantum Physics: Common Concepts
• Reconciling modern machine learning practice and the bias-variance trade-off
• Machine learning phases of matter
• Noise Dressing of Financial Correlation Matrices
• NSFnets (Navier-Stokes Flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
• Universal and non-universal properties of cross-correlations in financial time series
• A Random Matrix Approach to Cross-Correlations in Financial Data
• Scaling of the distribution of fluctuations of financial market indices
• Many-body quantum state tomography with neural networks
• Learning phase transitions by confusion
• Fast inference of deep neural networks in FPGAs for particle physics
• Deep Learning and its Application to LHC Physics
• Machine learning quantum phases of matter beyond the fermion sign problem
• Random Matrices and Chaos in Nuclear Physics
• Machine Learning Phases of Strongly Correlated Fermions
• Cleaning large correlation matrices: tools from random matrix theory
• Two models of double descent for weak features
• Identifying Quantum Phase Transitions with Adversarial Neural Networks
• Random Matrices and Chaos in Nuclear Spectra
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Level statistics and localization transitions of Lévy matrices
• Portfolio Optimization and the Random Magnet Problem
• Spectral correlations in the crossover between GUE and Poisson regularity: on the identification of scales
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Physics-informed generative neural network: an application to troposphere temperature prediction
• The Spectrum of the two dimensional Hubbard model at low filling
• Supervised learning in Hamiltonian reconstruction from local measurements on eigenstates

Cited by in corpus (5)

• Universal characteristics of deep neural network loss surfaces from random matrix theory
• Eigenvalue spectral tails and localization properties of asymmetric networks
• Boundary between noise and information applied to filtering neural network weight matrices
• Randomness and signal propagation in physics-informed neural networks (PINNs): A neural PDE perspective
• Task complexity shapes internal representations and robustness in neural networks