Global Convergence of Three-layer Neural Networks in the Mean Field Regime

Abstract

In the mean field regime, neural networks are appropriately scaled so that as the width tends to infinity, the learning dynamics tends to a nonlinear and nontrivial dynamical limit, known as the mean field limit. This lends a way to study large-width neural networks via analyzing the mean field limit. Recent works have successfully applied such analysis to two-layer networks and provided global convergence guarantees. The extension to multilayer ones however has been a highly challenging puzzle, and little is known about the optimization efficiency in the mean field regime when there are more than two layers. In this work, we prove a global convergence result for unregularized feedforward three-layer networks in the mean field regime. We first develop a rigorous framework to establish the mean field limit of three-layer networks under stochastic gradient descent training. To that end, we propose the idea of a \textit{neuronal embedding}, which comprises of a fixed probability space that encapsulates neural networks of arbitrary sizes. The identified mean field limit is then used to prove a global convergence guarantee under suitable regularity and convergence mode assumptions, which -- unlike previous works on two-layer networks -- does not rely critically on convexity. Underlying the result is a universal approximation property, natural of neural networks, which importantly is shown to hold at \textit{any} finite training time (not necessarily at convergence) via an algebraic topology argument.

Appear in ICLR 2021. This is the conference version of arXiv:2001.11443 (which contains treatment of the multilayer neural nets and their global convergence)

References in corpus (7)

• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• A mean-field limit for certain deep neural networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Stochastic Particle Gradient Descent for Infinite Ensembles
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime

Cited by in corpus (1)

• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis