Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation

Abstract

Several recent trends in machine learning theory and practice, from the design of state-of-the-art Gaussian Process to the convergence analysis of deep neural nets (DNNs) under stochastic gradient descent (SGD), have found it fruitful to study wide random neural networks. Central to these approaches are certain scaling limits of such networks. We unify these results by introducing a notion of a straightline \emph{tensor program} that can express most neural network computations, and we characterize its scaling limit when its tensors are large and randomized. From our framework follows (1) the convergence of random neural networks to Gaussian processes for architectures such as recurrent neural networks, convolutional neural networks, residual networks, attention, and any combination thereof, with or without batch normalization; (2) conditions under which the \emph{gradient independence assumption} -- that weights in backpropagation can be assumed to be independent from weights in the forward pass -- leads to correct computation of gradient dynamics, and corrections when it does not; (3) the convergence of the Neural Tangent Kernel, a recently proposed kernel used to predict training dynamics of neural networks under gradient descent, at initialization for all architectures in (1) without batch normalization. Mathematically, our framework is general enough to rederive classical random matrix results such as the semicircle and the Marchenko-Pastur laws, as well as recent results in neural network Jacobian singular values. We hope our work opens a way toward design of even stronger Gaussian Processes, initialization schemes to avoid gradient explosion/vanishing, and deeper understanding of SGD dynamics in modern architectures.

arXiv commenttldr: A theoretical tool for understanding the behavior of large width randomly initialized neural networks for almost all deep learning architectures. For a gentler introduction to Gaussian Process results here and several extensions, we recommend the reader to look at arXiv:1910.12478

References in corpus (32)

• Message Passing Algorithms for Compressed Sensing
• Densely Connected Convolutional Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Stimulus-Dependent Suppression of Chaos in Recurrent Neural Networks
• Exponential expressivity in deep neural networks through transient chaos
• Compressive Phase Retrieval via Generalized Approximate Message Passing
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Entropy and mutual information in models of deep neural networks
• Adversarial Examples, Uncertainty, and Transfer Testing Robustness in Gaussian Process Hybrid Deep Networks
• On the Convergence Rate of Training Recurrent Neural Networks
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Dynamics of Random Neural Networks with Bistable Units
• Randomized Prior Functions for Deep Reinforcement Learning
• Stochastic Variational Deep Kernel Learning
• Path Integral Approach to Random Neural Networks
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• A Mean Field Theory of Batch Normalization
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• How to Start Training: The Effect of Initialization and Architecture
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Exploring the Function Space of Deep-Learning Machines
• Deep Gaussian Processes with Convolutional Kernels
• A Gaussian Process perspective on Convolutional Neural Networks
• Inference in Deep Networks in High Dimensions

Cited by in corpus (109)

• On Exact Computation with an Infinitely Wide Neural Net
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Explaining Neural Scaling Laws
• Optimization for deep learning: theory and algorithms
• The Modern Mathematics of Deep Learning
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• A Theoretical Analysis of Deep Q-Learning
• When and why PINNs fail to train: A neural tangent kernel perspective
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Enhanced Convolutional Neural Tangent Kernels
• Neural Networks and Quantum Field Theory
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Kernel and Rich Regimes in Overparametrized Models
• A Mean Field Theory of Batch Normalization
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Asymptotics of Wide Networks from Feynman Diagrams
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Finite Versus Infinite Neural Networks: an Empirical Study
• A Fine-Grained Spectral Perspective on Neural Networks
• Nonperturbative renormalization for the neural network-QFT correspondence
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Mathematical Models of Overparameterized Neural Networks
• Asymptotics of representation learning in finite Bayesian neural networks
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Feature Learning in Infinite-Width Neural Networks
• Tensor Programs III: Neural Matrix Laws
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Asymptotics of Wide Convolutional Neural Networks
• Towards NNGP-guided Neural Architecture Search
• Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• On the infinite width limit of neural networks with a standard parameterization
• Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks
• Robust Pruning at Initialization
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Generalization bounds for deep learning
• Neural networks are a priori biased towards Boolean functions with low entropy
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Exact marginal prior distributions of finite Bayesian neural networks
• Stable behaviour of infinitely wide deep neural networks
• The Local Elasticity of Neural Networks
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Infinite Neural Network Quantum States: Entanglement and Training Dynamics
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• Is SGD a Bayesian sampler? Well, almost
• Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Conditional Temporal Neural Processes with Covariance Loss
• On Infinite-Width Hypernetworks
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Richer priors for infinitely wide multi-layer perceptrons
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization
• The Recurrent Neural Tangent Kernel
• Exact posterior distributions of wide Bayesian neural networks
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Non-asymptotic approximations of neural networks by Gaussian processes
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• A Deep Conditioning Treatment of Neural Networks
• Infinitely Wide Tensor Networks as Gaussian Process
• On Random Kernels of Residual Architectures
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Implicit bias of deep linear networks in the large learning rate phase
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World Distributions
• Kernel-Based Smoothness Analysis of Residual Networks
• Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Scaling Neural Tangent Kernels via Sketching and Random Features
• Implicit Bias of Linear RNNs
• A Temporal Kernel Approach for Deep Learning with Continuous-time Information
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• Fast Adaptation with Linearized Neural Networks
• Making Method of Moments Great Again? -- How can GANs learn distributions
• Properties of the After Kernel
• The Renyi Gaussian Process: Towards Improved Generalization
• Quantifying Epistemic Uncertainty in Deep Learning
• Phase Diagram of Initial Condensation for Two-layer Neural Networks
• Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations
• MimicNorm: Weight Mean and Last BN Layer Mimic the Dynamic of Batch Normalization
• Experiments with Rich Regime Training for Deep Learning
• Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
• A note on regularised NTK dynamics with an application to PAC-Bayesian training
• Analytic expressions for the output evolution of a deep neural network
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
• On the Impact of Stable Ranks in Deep Nets
• Learning with Neural Tangent Kernels in Near Input Sparsity Time
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum