Bayesian RG Flow in Neural Network Field Theories

Abstract

The Neural Network Field Theory correspondence (NNFT) is a mapping from neural network (NN) architectures into the space of statistical field theories (SFTs). The Bayesian renormalization group (BRG) is an information-theoretic coarse graining scheme that generalizes the principles of the exact renormalization group (ERG) to arbitrarily parameterized probability distributions, including those of NNs. In BRG, coarse graining is performed in parameter space with respect to an information-theoretic distinguishability scale set by the Fisher information metric. In this paper, we unify NNFT and BRG to form a powerful new framework for exploring the space of NNs and SFTs, which we coin BRG-NNFT. With BRG-NNFT, NN training dynamics can be interpreted as inducing a flow in the space of SFTs from the information-theoretic `IR' $\to$ `UV'. Conversely, applying an information-shell coarse graining to the trained network's parameters induces a flow in the space of SFTs from the information-theoretic `UV' $\to$ `IR'. When the information-theoretic cutoff scale coincides with a standard momentum scale, BRG is equivalent to ERG. We demonstrate the BRG-NNFT correspondence on two analytically tractable examples. First, we construct BRG flows for trained, infinite-width NNs, of arbitrary depth, with generic activation functions. As a special case, we then restrict to architectures with a single infinitely-wide layer, scalar outputs, and generalized cos-net activations. In this case, we show that BRG coarse-graining corresponds exactly to the momentum-shell ERG flow of a free scalar SFT. Our analytic results are corroborated by a numerical experiment in which an ensemble of asymptotically wide NNs are trained and subsequently renormalized using an information-shell BRG scheme.

41 pages, 9 figures, 2 tables; clarified parts of text, fixed typos, added a pedagogical Bayesian inference example to section 2.1

References in corpus (46)

• Variational Inference: A Review for Statisticians
• Deep Learning using Rectified Linear Units (ReLU)
• Scaling Laws for Neural Language Models
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Deep Neural Networks as Gaussian Processes
• Exact Renormalization Group Equations. An Introductory Review
• Fundamentals of the Exact Renormalization Group
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• The Principles of Deep Learning Theory
• An elementary introduction to information geometry
• Mutual Information, Neural Networks and the Renormalization Group
• Explaining Neural Scaling Laws
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
• Neural Networks and Quantum Field Theory
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Asymptotics of Wide Networks from Feynman Diagrams
• Quantum field-theoretic machine learning
• Learning Lattice Quantum Field Theories with Equivariant Continuous Flows
• Nonperturbative renormalization for the neural network-QFT correspondence
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics
• Renormalization Group Flow as Optimal Transport
• Asymptotics of representation learning in finite Bayesian neural networks
• Optimal Renormalization Group Transformation from Information Theory
• Statistical physics through the lens of real-space mutual information
• Relevance in the Renormalization Group and in Information Theory
• Bayesian Renormalization
• Symmetries and phase diagrams with real-space mutual information neural estimation
• Finite size corrections for neural network Gaussian processes
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Applications of Machine Learning to Lattice Quantum Field Theory
• A Solvable Model of Neural Scaling Laws
• The Inverse of Exact Renormalization Group Flows as Statistical Inference
• Normalizing flows for lattice gauge theory in arbitrary space-time dimension
• Scaling and renormalization in high-dimensional regression
• Renormalization in the neural network-quantum field theory correspondence
• Criticality versus uniformity in deep neural networks
• Quantum field theories, Markov random fields and machine learning
• Renormalizing Diffusion Models
• Bayesian posterior approximation with stochastic ensembles
• Grokking modular arithmetic
• Machine learning with quantum field theories
• NCoder -- A Quantum Field Theory approach to encoding data
• Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics

Cited by in corpus (1)

• Wilsonian Renormalization of Neural Network Gaussian Processes