AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Abstract

Recurrent neural networks have gained widespread use in modeling sequential data. Learning long-term dependencies using these models remains difficult though, due to exploding or vanishing gradients. In this paper, we draw connections between recurrent networks and ordinary differential equations. A special form of recurrent networks called the AntisymmetricRNN is proposed under this theoretical framework, which is able to capture long-term dependencies thanks to the stability property of its underlying differential equation. Existing approaches to improving RNN trainability often incur significant computation overhead. In comparison, AntisymmetricRNN achieves the same goal by design. We showcase the advantage of this new architecture through extensive simulations and experiments. AntisymmetricRNN exhibits much more predictable dynamics. It outperforms regular LSTM models on tasks requiring long-term memory and matches the performance on tasks where short-term dependencies dominate despite being much simpler.

arXiv commentPublished as a conference paper at ICLR 2019

Cited by in corpus (41)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Normalizing Flows: An Introduction and Review of Current Methods
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• Dissecting Neural ODEs
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
• Symplectic Recurrent Neural Networks
• Lipschitz Recurrent Neural Networks
• Accelerating Neural ODEs Using Model Order Reduction
• Forecasting Sequential Data using Consistent Koopman Autoencoders
• Continuous-in-Depth Neural Networks
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• Stable Neural Flows
• UnICORNN: A recurrent model for learning very long time dependencies
• Bridging the Gap: Machine Learning to Resolve Improperly Modeled Dynamics
• Framing RNN as a kernel method: A neural ODE approach
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• Differentiable programming and its applications to dynamical systems
• Noisy Recurrent Neural Networks
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Stable Invariant Models via Koopman Spectra
• A non-autonomous equation discovery method for time signal classification
• RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
• R-FORCE: Robust Learning for Random Recurrent Neural Networks
• Interpolation between Residual and Non-Residual Networks
• Efficient and Accurate Gradients for Neural SDEs
• Time-Reversal Symmetric ODE Network
• Learning-Accelerated ADMM for Distributed Optimal Power Flow
• A Differential Geometry Perspective on Orthogonal Recurrent Models
• How Chaotic Are Recurrent Neural Networks?
• A unified framework for Hamiltonian deep neural networks
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Long Expressive Memory for Sequence Modeling
• Constructing Gradient Controllable Recurrent Neural Networks Using Hamiltonian Dynamics
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm
• Online Spatiotemporal Action Detection and Prediction via Causal Representations
• Review: Ordinary Differential Equations For Deep Learning