Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks
arXiv:1809.08587
Abstract
We study the dynamics of gradient descent on objective functions of the form (with respect to scalar parameters ), which arise in the context of training depth- linear neural networks. We prove that for standard random initializations, and under mild assumptions on , the number of iterations required for convergence scales exponentially with the depth . We also show empirically that this phenomenon can occur in higher dimensions, where each is a matrix. This highlights a potential obstacle in understanding the convergence of gradient-based methods for deep linear neural networks, where is large.
arXiv commentComparison to previous version: Fixed a bug in lemma 1 part 3 (does not affect any other part of the paper)
References in corpus (5)
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
- Gradient descent aligns the layers of deep linear networks
- Depth Creates No Bad Local Minima