Products of Many Large Random Matrices and Gradients in Deep Neural Networks

Abstract

We study products of random matrices in the regime where the number of terms and the size of the matrices simultaneously tend to infinity. Our main theorem is that the logarithm of the ${\ell }_2$ norm of such a product applied to any fixed vector is asymptotically Gaussian. The fluctuations we find can be thought of as a finite temperature correction to the limit in which first the size and then the number of matrices tend to infinity. Depending on the scaling limit considered, the mean and variance of the limiting Gaussian depend only on either the first two or the first four moments of the measure from which matrix entries are drawn. We also obtain explicit error bounds on the moments of the norm and the Kolmogorov-Smirnov distance to a Gaussian. Finally, we apply our result to obtain precise information about the stability of gradients in randomly initialized deep neural networks with ReLU activations. This provides a quantitative measure of the extent to which the exploding and vanishing gradient problem occurs in a fully connected neural network with ReLU activations and a given architecture.

arXiv commentv1. 26p. Comments Welcome

References in corpus (13)

• Black Holes and Random Matrices
• Recent exact and asymptotic results for products of independent random matrices
• The Emergence of Spectral Universality in Deep Networks
• Some Open Problems in Random Matrix Theory and the Theory of Integrable Systems. II
• Top-quark mass measurement in the all-hadronic $t\bar{t}$ decay channel at $\sqrt{s}$ = 8 TeV with the ATLAS detector
• Universal distribution of Lyapunov exponents for products of Ginibre matrices
• Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
• Lyapunov exponents for products of complex Gaussian random matrices
• From Integrable to Chaotic Systems: Universal Local Statistics of Lyapunov exponents
• On the largest Lyapunov exponent for products of Gaussian matrices
• Asymptotics of finite system Lyapunov exponents for some random matrix ensembles
• Lyapunov exponents for products of rectangular real, complex and quaternionic Ginibre matrices
• On Products of Random Matrices

Cited by in corpus (19)

• Asymptotics of Wide Networks from Feynman Diagrams
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Eigenvalue distribution of nonlinear models of random matrices
• Lyapunov exponent, universality and phase transition for products of random matrices
• Schwinger-Dyson and loop equations for a product of square Ginibre random matrices
• Deep ReLU Networks Preserve Expected Length
• Products of Complex Rectangular and Hermitian Random Matrices
• Precise characterization of the prior predictive distribution of deep ReLU networks
• Hard Edge Statistics of Products of Pólya Ensembles and Shifted GUE's
• Product Matrix Processes with Symplectic and Orthogonal Invariance via Symmetric Functions
• Phase transitions for infinite products of large non-Hermitian random matrices
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
• Fractional moment-preserving initialization schemes for training deep neural networks
• Beyond Folklore: A Scaling Calculus for the Design and Initialization of ReLU Networks
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Vanishing Curvature and the Power of Adaptive Methods in Randomly Initialized Deep Networks
• Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
• On the Expected Complexity of Maxout Networks