Neural Networks and Quantum Field Theory

Abstract

We propose a theoretical understanding of neural networks in terms of Wilsonian effective field theory. The correspondence relies on the fact that many asymptotic neural networks are drawn from Gaussian processes, the analog of non-interacting field theories. Moving away from the asymptotic limit yields a non-Gaussian process and corresponds to turning on particle interactions, allowing for the computation of correlation functions of neural network outputs with Feynman diagrams. Minimal non-Gaussian process likelihoods are determined by the most relevant non-Gaussian terms, according to the flow in their coefficients induced by the Wilsonian renormalization group. This yields a direct connection between overparameterization and simplicity of neural network likelihoods. Whether the coefficients are constants or functions may be understood in terms of GP limit symmetries, as expected from 't Hooft's technical naturalness. General theoretical calculations are matched to neural network experiments in the simplest class of models allowing the correspondence. Our formalism is valid for any of the many architectures that becomes a GP in an asymptotic limit, a property preserved under certain types of training.

arXiv commentv2: published in Machine Learning: Science and Technology. Additions include study of N-scaling, a correction for examples, and new experimental tests. 53 pages, 7 figures, and appendices

References in corpus (24)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Machine Translation by Jointly Learning to Align and Translate
• Semi-Supervised Classification with Graph Convolutional Networks
• Spectral Networks and Locally Connected Networks on Graphs
• Densely Connected Convolutional Networks
• Chameleon Cosmology
• Deep Convolutional Networks on Graph-Structured Data
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Layer Normalization
• Gated Graph Sequence Neural Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Deep Convolutional Networks as shallow Gaussian Processes
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Asymptotics of Wide Networks from Feynman Diagrams
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• A Fine-Grained Spectral Perspective on Neural Networks
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Feature Learning in Infinite-Width Neural Networks
• Finite size corrections for neural network Gaussian processes
• Tensor Programs III: Neural Matrix Laws
• Non-Gaussian processes and neural networks at finite widths

Cited by in corpus (42)

• Representation Learning via Quantum Neural Tangent Kernels
• Disorder Averaging and its UV (Dis)Contents
• The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
• Quantum field-theoretic machine learning
• Physics-Driven Learning for Inverse Problems in Quantum Chromodynamics
• Nonperturbative renormalization for the neural network-QFT correspondence
• Asymptotics of representation learning in finite Bayesian neural networks
• Contrasting random and learned features in deep Bayesian linear regression
• Initial value problem in string-inspired nonlocal field theory
• Machine Learning Lie Structures & Applications to Physics
• Bayesian Renormalization
• Unified field theoretical approach to deep and recurrent neuronal networks
• Field theoretical approach for signal detection in nearly continuous positive spectra II: Tensorial data
• p-Adic Statistical Field Theory and Deep Belief Networks
• Signal detection in nearly continuous spectra and symmetry breaking
• p-Adic Statistical Field Theory and Convolutional Deep Boltzmann Machines
• Laziness, Barren Plateau, and Noise in Machine Learning
• Exact marginal prior distributions of finite Bayesian neural networks
• Conformal Fields from Neural Networks
• Machine Learning Regularization for the Minimum Volume Formula of Toric Calabi-Yau 3-folds
• Metric Flows with Neural Networks
• Infinite Neural Network Quantum States: Entanglement and Training Dynamics
• Characterizing 4-string contact interaction using machine learning
• On learning higher-order cumulants in diffusion models
• Misanthropic Entropy and Renormalization as a Communication Channel
• Universes as Big Data
• Neural network representation of quantum systems
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Black holes and the loss landscape in machine learning
• Some Open Questions in Quiver Gauge Theory
• Generative AI for Brane Configurations and Coamoeba
• Bayesian RG Flow in Neural Network Field Theories
• NCoder -- A Quantum Field Theory approach to encoding data
• Functional renormalization group for signal detection and stochastic ergodicity breaking
• Wilsonian Renormalization of Neural Network Gaussian Processes
• Particle Physics Model Building with Reinforcement Learning
• Gauge-covariant stochastic neural fields: Stability and finite-width effects
• Functional Renormalization Group Approach for Signal Detection
• Row and column detection complexities of character tables
• The neural networks with tensor weights and emergent fermionic Wick rules in the large-width limit
• Conformal Defects in Neural Network Field Theories
• Functional Renormalization for Signal Detection: Dimensional Analysis and Dimensional Phase Transition for Nearly Continuous Spectra Effective Field Theory