Global Convergence of Gradient Descent for Deep Linear Residual Networks

Abstract

We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent converges to an $\epsilon$-optimal point in $O(L^3\log (1/\epsilon ))$ iterations, which scales polynomially with the network depth $L$. Our result and the $\exp (\Omega (L))$ convergence time for the standard initialization (Xavier or near-identity) [Shamir, 2018] together demonstrate the importance of the residual structure and the initialization in the optimization for deep linear neural networks, especially when $L$ is large.

References in corpus (8)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Gradient descent aligns the layers of deep linear networks
• Identity Matters in Deep Learning
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

Cited by in corpus (7)

• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Continuous vs. Discrete Optimization of Deep Neural Networks
• The loss landscape of deep linear neural networks: a second-order analysis
• Implicit regularization of deep residual networks towards neural ODEs
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network