Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks

Abstract

Recurrent neural networks have gained widespread use in modeling sequence data across various domains. While many successful recurrent architectures employ a notion of gating, the exact mechanism that enables such remarkable performance is not well understood. We develop a theory for signal propagation in recurrent networks after random initialization using a combination of mean field theory and random matrix theory. To simplify our discussion, we introduce a new RNN cell with a simple gating mechanism that we call the minimalRNN and compare it with vanilla RNNs. Our theory allows us to define a maximum timescale over which RNNs can remember an input. We show that this theory predicts trainability for both recurrent architectures. We show that gated recurrent networks feature a much broader, more robust, trainable region than vanilla RNNs, which corroborates recent experimental findings. Finally, we develop a closed-form critical initialization scheme that achieves dynamical isometry in both vanilla RNNs and minimalRNNs. We show that this results in significantly improvement in training dynamics. Finally, we demonstrate that the minimalRNN achieves comparable performance to its more complex counterparts, such as LSTMs or GRUs, on a language modeling task.

arXiv commentICML 2018 Conference Proceedings

Cited by in corpus (21)

• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Convergence Rate of Training Recurrent Neural Networks
• Cellular automata as convolutional neural networks
• Sorting out Lipschitz function approximation
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Feature Learning in Infinite-Width Neural Networks
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Tensor Programs III: Neural Matrix Laws
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Information Geometry of Orthogonal Initializations and Training
• Noisy Recurrent Neural Networks
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights