Asymptotics of Wide Networks from Feynman Diagrams

Abstract

Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dynamics, improving existing bounds and deriving new results on wide network evolution during stochastic gradient descent. Going beyond the strict large width limit, we present closed-form expressions for higher-order terms governing wide network training, and test these predictions empirically.

arXiv comment10 pages, 3 figures, 1 Table + Appendices

References in corpus (18)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling description of generalization with number of parameters in deep learning
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Gradient Descent Happens in a Tiny Subspace
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• SGD Learns the Conjugate Kernel Class of the Network
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• Products of Many Large Random Matrices and Gradients in Deep Neural Networks
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• A Correspondence Between Random Neural Networks and Statistical Field Theory
• Limitations of Lazy Training of Two-layers Neural Networks

Cited by in corpus (22)

• Explaining Neural Scaling Laws
• Representation Learning via Quantum Neural Tangent Kernels
• Neural Networks and Quantum Field Theory
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Nonperturbative renormalization for the neural network-QFT correspondence
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Asymptotics of representation learning in finite Bayesian neural networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• On the infinite width limit of neural networks with a standard parameterization
• Non-Gaussian processes and neural networks at finite widths
• Laziness, Barren Plateau, and Noise in Machine Learning
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• On the asymptotics of wide networks with polynomial activations
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Bayesian RG Flow in Neural Network Field Theories
• Dynamically Stable Infinite-Width Limits of Neural Classifiers
• Understanding Deflation Process in Over-parametrized Tensor Decomposition
• Dynamical transition in controllable quantum neural networks with large depth
• A Dynamical Central Limit Theorem for Shallow Neural Networks
• Catapult Dynamics and Phase Transitions in Quadratic Nets