Tensor Programs III: Neural Matrix Laws

Abstract

In a neural network (NN), *weight matrices* linearly transform inputs into *preactivations* that are then transformed nonlinearly into *activations*. A typical NN interleaves multitudes of such linear and nonlinear transforms to express complex functions. Thus, the (pre-)activations depend on the weights in an intricate manner. We show that, surprisingly, (pre-)activations of a randomly initialized NN become *independent* from the weights as the NN's widths tend to infinity, in the sense of asymptotic freeness in random matrix theory. We call this the Free Independence Principle (FIP), which has these consequences: 1) It rigorously justifies the calculation of asymptotic Jacobian singular value distribution of an NN in Pennington et al. [36,37], essential for training ultra-deep NNs [48]. 2) It gives a new justification of gradient independence assumption used for calculating the Neural Tangent Kernel of a neural network. FIP and these results hold for any neural architecture. We show FIP by proving a Master Theorem for any Tensor Program, as introduced in Yang [50,51], generalizing the Master Theorems proved in those works. As warmup demonstrations of this new Master Theorem, we give new proofs of the semicircle and Marchenko-Pastur laws, which benchmarks our framework against these fundamental mathematical results.

References in corpus (28)

• Message Passing Algorithms for Compressed Sensing
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• On the Convergence Rate of Training Recurrent Neural Networks
• The Emergence of Spectral Universality in Deep Networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• A Mean Field Theory of Batch Normalization
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• How to Start Training: The Effect of Initialization and Architecture
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Conductivity tensor of striped quantum Hall phases
• Infinite attention: NNGP and NTK for deep attention networks
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• The Recurrent Neural Tangent Kernel
• Spectrum concentration in deep residual learning: a free probability approach

Cited by in corpus (7)

• Neural Networks and Quantum Field Theory
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Nonperturbative renormalization for the neural network-QFT correspondence
• The Recurrent Neural Tangent Kernel
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Implicit Acceleration and Feature Learning in Infinitely Wide Neural Networks with Bottlenecks
• Regularization in ResNet with Stochastic Depth