Nonperturbative renormalization for the neural network-QFT correspondence

Abstract

In a recent work arXiv:2008.08601, Halverson, Maiti and Stoner proposed a description of neural networks in terms of a Wilsonian effective field theory. The infinite-width limit is mapped to a free field theory, while finite $N$ corrections are taken into account by interactions (non-Gaussian terms in the action). In this paper, we study two related aspects of this correspondence. First, we comment on the concepts of locality and power-counting in this context. Indeed, these usual space-time notions may not hold for neural networks (since inputs can be arbitrary), however, the renormalization group provides natural notions of locality and scaling. Moreover, we comment on several subtleties, for example, that data components may not have a permutation symmetry: in that case, we argue that random tensor field theories could provide a natural generalization. Second, we improve the perturbative Wilsonian renormalization from arXiv:2008.08601 by providing an analysis in terms of the nonperturbative renormalization group using the Wetterich-Morris equation. An important difference with usual nonperturbative RG analysis is that only the effective (IR) 2-point function is known, which requires setting the problem with care. Our aim is to provide a useful formalism to investigate neural networks behavior beyond the large-width limit (i.e.~far from Gaussian limit) in a nonperturbative fashion. A major result of our analysis is that changing the standard deviation of the neural network weight distribution can be interpreted as a renormalization flow in the space of networks. We focus on translations invariant kernels and provide preliminary numerical results.

63 pages, 13 figures (published version)

References in corpus (12)

• Deep Learning in Neural Networks: An Overview
• Exact evolution equation for the effective potential
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Non perturbative renormalization group and momentum dependence of n-point functions (II)
• Wilsonian Effective Action of Superstring Theory
• String Field Theory -- A Modern Introduction
• Asymptotics of Wide Networks from Feynman Diagrams
• Deep learning and the renormalization group
• Non-Perturbative Renormalization Group calculation of the scalar self-energy
• A Correspondence Between Random Neural Networks and Statistical Field Theory
• Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators
• Why Unsupervised Deep Networks Generalize

Cited by in corpus (17)

• Disorder Averaging and its UV (Dis)Contents
• Initial value problem in string-inspired nonlocal field theory
• p-Adic Statistical Field Theory and Deep Belief Networks
• A Triumvirate of AI Driven Theoretical Discovery
• p-Adic Statistical Field Theory and Convolutional Deep Boltzmann Machines
• Characterizing 4-string contact interaction using machine learning
• Misanthropic Entropy and Renormalization as a Communication Channel
• Black holes and the loss landscape in machine learning
• Bayesian RG Flow in Neural Network Field Theories
• Functional renormalization group for signal detection and stochastic ergodicity breaking
• Wilsonian Renormalization of Neural Network Gaussian Processes
• Gauge-covariant stochastic neural fields: Stability and finite-width effects
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
• Functional renormalization group for p=2 like glassy matrices in the planar approximation: I. Vertex expansion at equilibrium
• The neural networks with tensor weights and emergent fermionic Wick rules in the large-width limit
• Functional Renormalization Group Approach for Signal Detection
• Functional Renormalization for Signal Detection: Dimensional Analysis and Dimensional Phase Transition for Nearly Continuous Spectra Effective Field Theory