Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks

Abstract

In this paper, we investigate a two-layer fully connected neural network of the form $f(X)=\frac{1}{\sqrt{d_1}}{\mathbf{a}}^{\top }\sigma \left(WX\right)$, where $X\in {\mathbb{R}}^{d_0\times n}$ is a deterministic data matrix, $W\in {\mathbb{R}}^{d_1\times d_0}$ and $\mathbf{a}\in {\mathbb{R}}^{d_1}$ are random Gaussian weights, and $\sigma$ is a nonlinear activation function. We study the limiting spectral distributions of two empirical kernel matrices associated with $f(X)$: the empirical conjugate kernel (CK) and neural tangent kernel (NTK), beyond the linear-width regime ($d_1\asymp n$). We focus on the $\text{ultra-wide regime}$, where the width $d_1$ of the first layer is much larger than the sample size $n$. Under appropriate assumptions on $X$ and $\sigma$, a deformed semicircle law emerges as $d_1/n\to \infty$ and $n\to \infty$. We first prove this limiting law for generalized sample covariance matrices with some dependency. To specify it for our neural network model, we provide a nonlinear Hanson-Wright inequality that is suitable for neural networks with random weights and Lipschitz activation functions. We also demonstrate non-asymptotic concentrations of the empirical CK and NTK around their limiting kernels in the spectral norm, along with lower bounds on their smallest eigenvalues. As an application, we show that random feature regression induced by the empirical kernel achieves the same asymptotic performance as its limiting kernel regression under the ultra-wide regime. This allows us to calculate the asymptotic training and test errors for random feature regression using the corresponding kernel regression.

49 pages, 5 figures

References in corpus (11)

• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• Convergence of the largest eigenvalue of normalized sample covariance matrices when p and n both tend to infinity with their ratio converging to zero
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• Universality Laws for High-Dimensional Learning with Random Features
• CLT for linear spectral statistics of normalized sample covariance matrices with the dimension much larger than the sample size
• What causes the test error? Going beyond bias-variance via ANOVA
• Generalization error of random features and kernel methods: hypercontractivity and kernel matrix concentration
• On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths
• Analysis of One-Hidden-Layer Neural Networks via the Resolvent Method
• Asymptotic normality for eigenvalue statistics of a general sample covariance matrix when $p/n\to \infty$ and applications