Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Abstract

The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been proved about the concrete effects of different initialization schemes. In this work, we analyze the effect of initialization in deep linear networks, and provide for the first time a rigorous proof that drawing the initial weights from the orthogonal group speeds up convergence relative to the standard Gaussian initialization with iid weights. We show that for deep networks, the width needed for efficient convergence to a global minimum with orthogonal initializations is independent of the depth, whereas the width needed for efficient convergence with Gaussian initializations scales linearly in the depth. Our results demonstrate how the benefits of a good initialization can persist throughout learning, suggesting an explanation for the recent empirical successes found by initializing very deep non-linear networks according to the principle of dynamical isometry.

International Conference on Learning Representations (ICLR) 2020

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Full-Capacity Unitary Recurrent Neural Networks
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• The Emergence of Spectral Universality in Deep Networks
• Depth Creates No Bad Local Minima
• Identity Matters in Deep Learning
• Recurrent Orthogonal Networks and Long-Memory Tasks
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function

Cited by in corpus (13)

• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Finite Versus Infinite Neural Networks: an Empirical Study
• What causes the test error? Going beyond bias-variance via ANOVA
• Towards NNGP-guided Neural Architecture Search
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training
• Continuous vs. Discrete Optimization of Deep Neural Networks
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
• On the validity of kernel approximations for orthogonally-initialized neural networks
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network