Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics

Abstract

A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurrent connectivity matrices to be orthogonal or unitary. This ensures eigenvalues with unit norm and thus stable dynamics and training. However this comes at the cost of reduced expressivity due to the limited variety of orthogonal transformations. We propose a novel connectivity structure based on the Schur decomposition and a splitting of the Schur form into normal and non-normal parts. This allows to parametrize matrices with unit-norm eigenspectra without orthogonality constraints on eigenbases. The resulting architecture ensures access to a larger space of spectrally constrained matrices, of which orthogonal matrices are a subset. This crucial difference retains the stability advantages and training speed of orthogonal RNNs while enhancing expressivity, especially on tasks that require computations over ongoing input sequences.

References in corpus (9)

• On the difficulty of training Recurrent Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Nonnormal amplification in random balanced neuronal networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies

Cited by in corpus (7)

• Metastable dynamics of neural circuits and networks
• Forecasting Sequential Data using Consistent Koopman Autoencoders
• UnICORNN: A recurrent model for learning very long time dependencies
• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies
• Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems
• Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units