Initialization of ReLUs for Dynamical Isometry

Abstract

Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles provide theoretical insights into the space of candidate models of which one is selected during training. The results obtained so far rely on mean field approximations that assume infinite layer width and that study average squared signals. We derive the joint signal output distribution exactly, without mean field assumptions, for fully-connected networks with Gaussian weights and biases, and analyze deviations from the mean field results. For rectified linear units, we further discuss limitations of the standard initialization scheme, such as its lack of dynamical isometry, and propose a simple alternative that overcomes these by initial parameter sharing.

arXiv commentNeurIPS 2019

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• How transferable are features in deep neural networks?
• Deep Neural Networks as Gaussian Processes
• Exponential expressivity in deep neural networks through transient chaos
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• The Emergence of Spectral Universality in Deep Networks
• A Mean Field Theory of Batch Normalization
• Critical initialisation for deep signal propagation in noisy rectifier neural networks