Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs

Abstract

Training recurrent neural networks (RNNs) on long sequence tasks is plagued with difficulties arising from the exponential explosion or vanishing of signals as they propagate forward or backward through the network. Many techniques have been proposed to ameliorate these issues, including various algorithmic and architectural modifications. Two of the most successful RNN architectures, the LSTM and the GRU, do exhibit modest improvements over vanilla RNN cells, but they still suffer from instabilities when trained on very long sequences. In this work, we develop a mean field theory of signal propagation in LSTMs and GRUs that enables us to calculate the time scales for signal propagation as well as the spectral properties of the state-to-state Jacobians. By optimizing these quantities in terms of the initialization hyperparameters, we derive a novel initialization scheme that eliminates or reduces training instabilities. We demonstrate the efficacy of our initialization scheme on multiple sequence tasks, on which it enables successful training while a standard initialization either fails completely or is orders of magnitude slower. We also observe a beneficial effect on generalization performance using this new initialization.

References in corpus (11)

• On the difficulty of training Recurrent Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Full-Capacity Unitary Recurrent Neural Networks
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• The Emergence of Spectral Universality in Deep Networks
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Kronecker Recurrent Units

Cited by in corpus (13)

• ReZero is All You Need: Fast Convergence at Large Depth
• Optimization for deep learning: theory and algorithms
• A Mean Field Theory of Batch Normalization
• Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Feature Learning in Infinite-Width Neural Networks
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
• Deep Learning Theory through the Lens of Statistical Physics: A Topical Review