On the Global Convergence of Training Deep Linear ResNets

Abstract

We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets). We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss. Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets. Compared with the global convergence result of GD for training standard deep linear networks (Du & Hu 2019), our condition on the neural network width is sharper by a factor of $O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance matrix of the training data. We further propose a modified identity input and output transformations, and show that a $(d+k)$-wide neural network is sufficient to guarantee the global convergence of GD/SGD, where $d,k$ are the input and output dimensions respectively.

arXiv comment26 pages, 1 figure. In ICLR 2020

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Gradient descent aligns the layers of deep linear networks
• Depth Creates No Bad Local Minima
• SGD Learns the Conjugate Kernel Class of the Network
• When is a Convolutional Filter Easy To Learn?
• Learning One-hidden-layer ReLU Networks via Gradient Descent
• Topology and Geometry of Half-Rectified Network Optimization
• Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks
• Gradient Descent Finds Global Minima for Generalizable Deep Neural Networks of Practical Sizes

Cited by in corpus (4)

• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• Continuous vs. Discrete Optimization of Deep Neural Networks
• Understanding Dynamics of Nonlinear Representation Learning and Its Application
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum