Eigenvalue distribution of nonlinear models of random matrices

Abstract

This paper is concerned with the asymptotic empirical eigenvalue distribution of a non linear random matrix ensemble. More precisely we consider $M=\frac{1}{m}Y{Y}^{\ast }$ with $Y=f(WX)$ where $W$ and $X$ are random rectangular matrices with i.i.d. centered entries. The function $f$ is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where $W$ and $X$ have sub-Gaussian tails and $f$ is real analytic. This extends a previous result where the case of Gaussian matrices $W$ and $X$ is considered. We also investigate the same questions in the multi-layer case, regarding neural network applications.

36 pages, 19 figures. Paper shortened (the behavior of the largest eigenvalue is removed and postponed to another article after noticing an error)

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Machine learning and the physical sciences
• The Loss Surfaces of Multilayer Networks
• Entropy and mutual information in models of deep neural networks
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Raney distributions and random matrix theory
• Concentration of Measure and Large Random Matrices with an application to Sample Covariance Matrices

Cited by in corpus (4)

• Right large deviation principle for the top eigenvalue of the sum or product of invariant random matrices
• Structured random matrices and cyclic cumulants: A free probability approach
• Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks
• Distribution of singular values in large sample cross-covariance matrices