Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks

Abstract

We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks. In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality, we show that the eigenvalue distributions of the CK and NTK converge to deterministic limits. The limit for the CK is described by iterating the Marcenko-Pastur map across the hidden layers. The limit for the NTK is equivalent to that of a linear combination of the CK matrices across layers, and may be described by recursive fixed-point equations that extend this Marcenko-Pastur map. We demonstrate the agreement of these asymptotic predictions with the observed spectra for both synthetic and CIFAR-10 training data, and we perform a small simulation to investigate the evolutions of these spectra over training.

References in corpus (20)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Asymptotics of Wide Networks from Feynman Diagrams
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• A Fine-Grained Spectral Perspective on Neural Networks
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• The Dynamics of Learning: A Random Matrix Approach
• A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
• The asymptotic spectrum of the Hessian of DNN throughout training
• Restricted Isometry Property under High Correlations

Cited by in corpus (3)

• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent
• Largest Eigenvalues of the Conjugate Kernel of Single-Layered Neural Networks
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications