Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Abstract

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior.

References in corpus (19)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Opening the Black Box of Deep Neural Networks via Information
• Understanding Deep Convolutional Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Provable Bounds for Learning Some Deep Representations
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• The loss surface of deep and wide neural networks
• Scaling description of generalization with number of parameters in deep learning
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Deep Convolutional Networks as shallow Gaussian Processes
• Optimization Landscape and Expressivity of Deep CNNs
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Spurious Valleys in Two-layer Neural Network Optimization Landscapes
• Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?
• The loss landscape of overparameterized neural networks
• A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model

Cited by in corpus (22)

• Scaling description of generalization with number of parameters in deep learning
• Optimization for deep learning: theory and algorithms
• Cellular automata as convolutional neural networks
• When Do Neural Networks Outperform Kernel Methods?
• Dynamics of Deep Neural Networks and Neural Tangent Hierarchy
• A mean-field limit for certain deep neural networks
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Complexity Measures for Neural Networks with General Activation Functions Using Path-based Norms
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Theory III: Dynamics and Generalization in Deep Networks
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• An analytic theory of shallow networks dynamics for hinge loss classification
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
• A Note on the Global Convergence of Multilayer Neural Networks in the Mean Field Regime
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks