Deep Learning Theory through the Lens of Statistical Physics: A Topical Review

Abstract

The recent success of foundation models and deep learning has far outpaced our theoretical understanding of how large-scale neural networks learn and generalize. Statistical physics provides a natural language for describing such high-dimensional, nonlinear, and strongly coupled systems, offering concepts such as mean-field theory, phase transitions, energy landscapes, and stochastic dynamics. In this topical review, we adopt a statistical-physics way of thinking -- from concepts including phases and criticality, and free-energy viewpoints, to methods such as kernel and Gaussian-process limits, mean-field and dynamical mean-field theories, random-matrix tools, and Fokker-Planck formalisms, and to a mindset emphasizing universality, scaling, and tractable limits with explicit validity regimes. We survey recent progress on understanding modern deep learning through this lens, including mean-field analyses of wide networks, spectral and geometric perspectives on loss landscapes, stochastic-gradient-based training viewed as a dynamical process, and phase-transition analogies that organize phenomena such as overparameterization, generalization, and representation change. Our goal is to synthesize these threads into a coherent framework that clarifies what each approximation explains, when it is expected to hold, and how apparently competing narratives relate or diverge. Together, these perspectives form a unified physical framework that views representation learning as a collective phenomenon in high-dimensional systems, bridging deterministic and stochastic regimes of gradient-based optimization.

arXiv comment22 pages, 5 figures, 1 table. Topical review published in Front. Phys. 22(1), 011302 (2027); this version contains minor corrections

References in corpus (23)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Scaling Laws for Neural Language Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• Three Factors Influencing Minima in SGD
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Depth Creates No Bad Local Minima
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• On the Origin of Implicit Regularization in Stochastic Gradient Descent
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Stochastic Particle Gradient Descent for Infinite Ensembles
• Deep Information Propagation
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Mean-Field Langevin Dynamics: Exponential Convergence and Annealing
• Symmetry Induces Structure and Constraint of Learning
• Dynamical Mean-Field Theory of Self-Attention Neural Networks
• Precise Dynamics of Diagonal Linear Networks: A Unifying Analysis by Dynamical Mean-Field Theory
• Phase Transitions for Feature Learning in Neural Networks
• Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model