Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks

Abstract

We study the dynamics of gradient descent on objective functions of the form $f({\prod }_{i=1}^k{w}_i)$ (with respect to scalar parameters $w_1,\dots ,w_k$), which arise in the context of training depth-$k$ linear neural networks. We prove that for standard random initializations, and under mild assumptions on $f$, the number of iterations required for convergence scales exponentially with the depth $k$. We also show empirically that this phenomenon can occur in higher dimensions, where each $w_i$ is a matrix. This highlights a potential obstacle in understanding the convergence of gradient-based methods for deep linear neural networks, where $k$ is large.

Comparison to previous version: Fixed a bug in lemma 1 part 3 (does not affect any other part of the paper)

References in corpus (2)

• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Gradient descent aligns the layers of deep linear networks

Cited by in corpus (2)

• Channel Normalization in Convolutional Neural Network avoids Vanishing Gradients
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization