Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

Abstract

We consider learning two layer neural networks using stochastic gradient descent. The mean-field description of this learning dynamics approximates the evolution of the network weights by an evolution in the space of probability distributions in $R^D$ (where $D$ is the number of parameters associated to each neuron). This evolution can be defined through a partial differential equation or, equivalently, as the gradient flow in the Wasserstein space of probability distributions. Earlier work shows that (under some regularity assumptions), the mean field description is accurate as soon as the number of hidden units is much larger than the dimension $D$. In this paper we establish stronger and more general approximation guarantees. First of all, we show that the number of hidden units only needs to be larger than a quantity dependent on the regularity properties of the data, and independent of the dimensions. Next, we generalize this analysis to the case of unbounded activation functions, which was not covered by earlier bounds. We extend our results to noisy stochastic gradient descent. Finally, we show that kernel ridge regression can be recovered as a special limit of the mean field analysis.

arXiv comment61 pages

References in corpus (6)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Scaling description of generalization with number of parameters in deep learning
• Stochastic Particle Gradient Descent for Infinite Ensembles

Cited by in corpus (41)

• The generalization error of random features regression: Precise asymptotics and double descent curve
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• A Selective Overview of Deep Learning
• A mean-field limit for certain deep neural networks
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Mathematical Models of Overparameterized Neural Networks
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Directional convergence and alignment in deep learning
• On the infinite width limit of neural networks with a standard parameterization
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Functional renormalization group for multilinear disordered Langevin dynamics I: Formalism and first numerical investigations at equilibrium
• Gradients as Features for Deep Representation Learning
• Mean-field Langevin System, Optimal Control and Deep Neural Networks
• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• The asymptotic spectrum of the Hessian of DNN throughout training
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• Learning time-scales in two-layers neural networks
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Analysis of Knowledge Transfer in Kernel Regime
• Law of large numbers and central limit theorem for wide two-layer neural networks: the mini-batch and noisy case
• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
• A Chain Graph Interpretation of Real-World Neural Networks
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• A Mean-Field Theory for Kernel Alignment with Random Features in Generative and Discriminative Models
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Law of Large Numbers for Bayesian two-layer Neural Network trained with Variational Inference
• Decomposed resolution of finite-state aggregative optimal control problems
• Wasserstein Flow Meets Replicator Dynamics: A Mean-Field Analysis of Representation Learning in Actor-Critic
• A note on regularised NTK dynamics with an application to PAC-Bayesian training
• One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks
• Volumization as a Natural Generalization of Weight Decay
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
• A Mean-Field Theory for Learning the Schönberg Measure of Radial Basis Functions
• Implicit Compressibility of Overparametrized Neural Networks Trained with Heavy-Tailed SGD