Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Abstract

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

References in corpus (2)

• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Recovery Guarantees for One-hidden-layer Neural Networks

Cited by in corpus (9)

• Optimization for deep learning: theory and algorithms
• A mean-field limit for certain deep neural networks
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• A Note on the Global Convergence of Multilayer Neural Networks in the Mean Field Regime