On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime

Abstract

Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first-order methods nevertheless find the global optimum in the overparameterized regime. We study this phenomenon with mean-field analysis, by translating the training process of ResNet to a gradient-flow partial differential equation (PDE) and examining the convergence properties of this limiting process. The activation function is assumed to be $2$-homogeneous or partially $1$-homogeneous; the regularized ReLU satisfies the latter condition. We show that if the ResNet is sufficiently large, with depth and width depending algebraically on the accuracy and confidence levels, first-order optimization methods can find global minimizers that fit the training data.

arXiv commentarXiv admin note: text overlap with arXiv:2105.14417

References in corpus (18)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• The loss surface of deep and wide neural networks
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Gradient Descent Can Take Exponential Time to Escape Saddle Points
• Optimization Landscape and Expressivity of Deep CNNs
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?