A mean-field limit for certain deep neural networks

Abstract

Understanding deep neural networks (DNNs) is a key challenge in the theory of machine learning, with potential applications to the many fields where DNNs have been successfully used. This article presents a scaling limit for a DNN being trained by stochastic gradient descent. Our networks have a fixed (but arbitrary) number $L\ge 2$ of inner layers; $N\gg 1$ neurons per layer; full connections between layers; and fixed weights (or "random features" that are not trained) near the input and output. Our results describe the evolution of the DNN during training in the limit when $N\to +\infty$, which we relate to a mean field model of McKean-Vlasov type. Specifically, we show that network weights are approximated by certain "ideal particles" whose distribution and dependencies are described by the mean-field model. A key part of the proof is to show existence and uniqueness for our McKean-Vlasov problem, which does not seem to be amenable to existing theory. Our paper extends previous work on the $L=1$ case by Mei, Montanari and Nguyen; Rotskoff and Vanden-Eijnden; and Sirignano and Spiliopoulos. We also complement recent independent work on $L>1$ by Sirignano and Spiliopoulos (who consider a less natural scaling limit) and Nguyen (who nonrigorously derives similar results).

arXiv comment79 pages and 2 figures

References in corpus (9)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Scaling description of generalization with number of parameters in deep learning
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Cited by in corpus (22)

• Optimization for deep learning: theory and algorithms
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Learning Mean-Field Equations from Particle Data Using WSINDy
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• Mathematical Models of Overparameterized Neural Networks
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network
• Approximation power of random neural networks
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
• Gradient flows on graphons: existence, convergence, continuity equations
• An analytic theory of shallow networks dynamics for hinge loss classification
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• A Note on the Global Convergence of Multilayer Neural Networks in the Mean Field Regime
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks