Global Convergence of SGD On Two Layer Neural Nets

Abstract

In this note, we consider appropriately regularized ${\ell }_2-$empirical risk of depth $2$ nets with any number of gates and show bounds on how the empirical loss evolves for SGD iterates on it -- for arbitrary data and if the activation is adequately smooth and bounded like sigmoid and tanh. This in turn leads to a proof of global convergence of SGD for a special class of initializations. We also prove an exponentially fast convergence rate for continuous time SGD that also applies to smooth unbounded activations like SoftPlus. Our key idea is to show the existence of Frobenius norm regularized loss functions on constant-sized neural nets which are "Villani functions" and thus be able to build on recent progress with analyzing SGD on such objectives. Most critically the amount of regularization required for our analysis is independent of the size of the net.

22 pages, 6 figures. v3: Accepted at Information and Inference: A Journal of the IMA

References in corpus (17)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• On the Convergence Rate of Training Recurrent Neural Networks
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Enhanced Convolutional Neural Tangent Kernels
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• An Improved Analysis of Training Over-parameterized Deep Neural Networks
• Theory of Deep Learning IIb: Optimization Properties of SGD
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Convergence of gradient descent for deep neural networks
• Function approximation by neural nets in the mean-field regime: Entropic regularization and controlled McKean-Vlasov dynamics
• Edge of chaos as a guiding principle for modern neural network training
• Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions
• Quadratic models for understanding catapult dynamics of neural networks