Dynamical Regimes of Diffusion Models

Abstract

Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.

22 pages, 11 figures

References in corpus (7)

• Theoretical perspective on the glass transition and amorphous materials
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Score-Based Generative Modeling through Stochastic Differential Equations
• The Exponential Capacity of Dense Associative Memories
• Generative diffusion in very large dimensions
• Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
• Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Cited by in corpus (15)

• Nonreciprocal Spin-Glass Transition and Aging
• Nonequilbrium physics of generative diffusion models
• Speed-accuracy relations for diffusion models: Wisdom from nonequilibrium thermodynamics and optimal transport
• A Very Effective and Simple Diffusion Reconstruction for the Diluted Ising Model
• Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise
• Analysis of Diffusion Models for Manifold Data
• Harmonic Path Integral Diffusion
• Points as Tori: Fast Pointwise Signed Distance for Point Clouds
• Dreaming up scale invariance via inverse renormalization group
• Geometric Regularity in Deterministic Sampling Dynamics of Diffusion-based Generative Models
• U-Turn Diffusion
• The Information Dynamics of Generative Diffusion
• Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling
• Dynamical Regimes of Discrete Diffusion Models
• Convergence properties of Markov models for image generation with applications to spin-flip dynamics and to diffusion processes