UnICORNN: A recurrent model for learning very long time dependencies
arXiv:2103.05487
Abstract
The design of recurrent neural networks (RNNs) to accurately process sequential inputs with long-time dependencies is very challenging on account of the exploding and vanishing gradient problem. To overcome this, we propose a novel RNN architecture which is based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations that models networks of oscillators. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradient problem. A suite of experiments are presented to demonstrate that the proposed RNN provides state of the art performance on a variety of learning tasks with (very) long-time dependencies.
References in corpus (16)
- On the difficulty of training Recurrent Neural Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Hamiltonian Neural Networks
- Normalizing Flows for Probabilistic Modeling and Inference
- Full-Capacity Unitary Recurrent Neural Networks
- The UEA multivariate time series classification archive, 2018
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
- Optimizing Performance of Recurrent Neural Networks on GPUs
- Lipschitz Recurrent Neural Networks
- Deep Independently Recurrent Neural Network (IndRNN)
- Neural Rough Differential Equations for Long Time Series
- A recurrent neural network without chaos
- Reversible Recurrent Neural Networks
- Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
- Monash University, UEA, UCR Time Series Extrinsic Regression Archive