Random Walk Initialization for Training Very Deep Feedforward Networks

Abstract

Training very deep networks is an important open problem in machine learning. One of many difficulties is that the norm of the back-propagated error gradient can grow or decay exponentially. Here we show that training very deep feed-forward networks (FFNs) is not as difficult as previously thought. Unlike when back-propagation is applied to a recurrent network, application to an FFN amounts to multiplying the error gradient by a different random matrix at each layer. We show that the successive application of correctly scaled random matrices to an initial vector results in a random walk of the log of the norm of the resulting vectors, and we compute the scaling that makes this walk unbiased. The variance of the random walk grows only linearly with network depth and is inversely proportional to the size of each layer. Practically, this implies a gradient whose log-norm scales with the square root of the network depth and shows that the vanishing gradient problem can be mitigated by increasing the width of the layers. Mathematical analyses and experimental results using stochastic gradient descent to optimize tasks related to the MNIST and TIMIT datasets are provided to support these claims. Equations for the optimal matrix scaling are provided for the linear and ReLU cases.

10 pages, 4 figures

References in corpus (2)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• On the saddle point problem for non-convex optimization

Cited by in corpus (20)

• Training Very Deep Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• All you need is a good init
• Data-dependent Initializations of Convolutional Neural Networks
• Review: Deep Learning in Electron Microscopy
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Efficient Summarization with Read-Again and Copy Mechanism
• On the Potential of Simple Framewise Approaches to Piano Transcription
• Improving performance of recurrent neural network with relu nonlinearity
• Learning Combinations of Activation Functions
• Long short-term memory and learning-to-learn in networks of spiking neurons
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Learning to Generate Chairs, Tables and Cars with Convolutional Networks
• A graph-based probabilistic geometric deep learning framework with online enforcement of physical constraints to predict the criticality of defects in porous materials
• Automated Copper Alloy Grain Size Evaluation Using a Deep-learning CNN
• Training Feedforward Neural Networks with Standard Logistic Activations is Feasible
• Data-driven Weight Initialization with Sylvester Solvers
• Goldilocks Neural Networks
• Implementation of Optical Deep Neural Networks using the Fabry-Perot Interferometer