A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth

Abstract

Training deep neural networks with stochastic gradient descent (SGD) can often achieve zero training loss on real-world tasks although the optimization landscape is known to be highly non-convex. To understand the success of SGD for training deep neural networks, this work presents a mean-field analysis of deep residual networks, based on a line of works that interpret the continuum limit of the deep residual network as an ordinary differential equation when the network capacity tends to infinity. Specifically, we propose a new continuum limit of deep residual networks, which enjoys a good landscape in the sense that every local minimizer is global. This characterization enables us to derive the first global convergence result for multilayer neural networks in the mean-field regime. Furthermore, without assuming the convexity of the loss landscape, our proof relies on a zero-loss assumption at the global minimizer that can be achieved when the model shares a universal approximation property. Key to our result is the observation that a deep residual network resembles a shallow network ensemble, i.e. a two-layer network. We bound the difference between the shallow network and our ResNet model via the adjoint sensitivity method, which enables us to apply existing mean-field analyses of two-layer networks to deep networks. Furthermore, we propose several novel training schemes based on the new continuous model, including one training procedure that switches the order of the residual blocks and results in strong empirical performance on the benchmark datasets.

References in corpus (25)

• A Convergence Theory for Deep Learning via Over-Parameterization
• On Exact Computation with an Infinitely Wide Neural Net
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations
• Scalable Gradients for Stochastic Differential Equations
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Augmented Neural ODEs
• An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks
• A Priori Estimates of the Population Risk for Residual Networks
• A mean-field limit for certain deep neural networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Mean-Field Langevin Dynamics and Energy Landscape of Neural Networks
• Stochastic Particle Gradient Descent for Infinite Ensembles
• Dynamically Unfolding Recurrent Restorer: A Moving Endpoint Control Method for Image Restoration
• Towards Understanding the Importance of Shortcut Connections in Residual Networks
• Scaling the Scattering Transform: Deep Hybrid Networks
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Limitations of Lazy Training of Two-layers Neural Networks
• Global Convergence of Gradient Descent for Deep Linear Residual Networks

Cited by in corpus (17)

• Game-Theoretic Multiagent Reinforcement Learning
• Neural Network Approximation: Three Hidden Layers Are Enough
• Two-Layer Neural Networks for Partial Differential Equations: Optimization and Generalization Theory
• Large-time asymptotics in deep learning
• Analysis of feature learning in weight-tied autoencoders via the mean field lens
• ResNet After All? Neural ODEs and Their Numerical Solution
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
• Ergodicity of the underdamped mean-field Langevin dynamics
• Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Subquadratic Overparameterization for Shallow Neural Networks
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Stationary Density Estimation of Itô Diffusions Using Deep Learning
• Dynamic Game Theoretic Neural Optimizer