On learning higher-order cumulants in diffusion models

Abstract

To analyse how diffusion models learn correlations beyond Gaussian ones, we study the behaviour of higher-order cumulants, or connected n-point functions, under both the forward and backward process. We derive explicit expressions for the moment- and cumulant-generating functionals, in terms of the distribution of the initial data and properties of forward process. It is shown analytically that during the forward process higher-order cumulants are conserved in models without a drift, such as the variance-expanding scheme, and that therefore the endpoint of the forward process maintains nontrivial correlations. We demonstrate that since these correlations are encoded in the score function, higher-order cumulants are learnt in the backward process, also when starting from a normal prior. We confirm our analytical results in an exactly solvable toy model with nonzero cumulants and in scalar lattice field theory.

21 pages, many figures. Extended version of contribution awarded "best 'physics for AI' paper award" in the NeurIPS 2024 workshop "Machine Learning and the Physical Sciences"; v2: references and minor clarifications added, version to appear in Machine Learning: Science and Technology

References in corpus (17)

• Critical slowing down and error analysis in lattice QCD simulations
• Flow-based generative models for Markov chain Monte Carlo in lattice field theory
• Equivariant flow-based sampling for lattice gauge theory
• Estimation of Thermodynamic Observables in Lattice Field Theories with Deep Generative Models
• Asymptotically unbiased estimation of physical observables with neural samplers
• Deep Learning and AdS/CFT
• Neural Networks and Quantum Field Theory
• Quantum field-theoretic machine learning
• Learning Lattice Quantum Field Theories with Equivariant Continuous Flows
• Advances in machine-learning-based sampling motivated by lattice quantum chromodynamics
• Diffusion Models as Stochastic Quantization in Lattice Field Theory
• Bayesian Renormalization
• Sampling the lattice Nambu-Goto string using Continuous Normalizing Flows
• The Inverse of Exact Renormalization Group Flows as Statistical Inference
• Neural network representation of quantum systems
• Scaling Up Machine Learning For Quantum Field Theory with Equivariant Continuous Flows
• Stochastic weight matrix dynamics during learning and Dyson Brownian motion

Cited by in corpus (5)

• Scaling of Stochastic Normalizing Flows in $\mathrm{SU}(3)$ lattice gauge theory
• Scaling flow-based approaches for topology sampling in $\mathrm{SU}(3)$ gauge theory
• The neural networks with tensor weights and emergent fermionic Wick rules in the large-width limit
• Combining complex Langevin dynamics with score-based and energy-based diffusion models
• Journey from the Wilson exact RG towards the Wegner-Morris Fokker-Planck RG and the Carosso field-coarsening via Langevin stochastic processes