Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Abstract

Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with training by introducing various types of gating units into the architecture. While these gates empirically improve performance, how the addition of gates influences the dynamics and trainability of GRUs and LSTMs is not well understood. Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates. We leverage tools from random matrix theory and mean-field theory to study the state-to-state Jacobians of GRUs and LSTMs. We show that the update gate in the GRU and the forget gate in the LSTM can lead to an accumulation of slow modes in the dynamics. Moreover, the GRU update gate can poise the system at a marginally stable point. The reset gate in the GRU and the output and input gates in the LSTM control the spectral radius of the Jacobian, and the GRU reset gate also modulates the complexity of the landscape of fixed-points. Furthermore, for the GRU we obtain a phase diagram describing the statistical properties of fixed-points. We also provide a preliminary comparison of training performance to the various dynamical regimes realized by varying hyperparameters. Looking to the future, we have introduced a powerful set of techniques which can be adapted to a broad class of RNNs, to study the influence of various architectural choices on dynamics, and potentially motivate the principled discovery of novel architectures.

arXiv comment18+18 pages, 4 figures, to appear in Proceedings of Machine Learning Research Vol. 107, 2020, 1st Annual Conference on Mathematical and Scientific Machine Learning

References in corpus (12)

• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Skip-Thought Vectors
• Deep Neural Networks as Gaussian Processes
• Transition to chaos in random networks with cell-type-specific connectivity
• Dynamics of Random Neural Networks with Bistable Units
• Correlations between synapses in pairs of neurons slow down dynamics in randomly connected neural networks
• Statistical properties of eigenvectors in non-Hermitian Gaussian random matrix ensembles
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Eigenvalues of non-hermitian random matrices and Brown measure of non-normal operators: hermitian reduction and linearization method
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs

Cited by in corpus (2)

• Searching for long time scales without fine tuning
• Reverse engineering learned optimizers reveals known and novel mechanisms