Lipschitz Recurrent Neural Networks

Abstract

Viewing recurrent neural networks (RNNs) as continuous-time dynamical systems, we propose a recurrent unit that describes the hidden state's evolution with two parts: a well-understood linear component plus a Lipschitz nonlinearity. This particular functional form facilitates stability analysis of the long-term behavior of the recurrent unit using tools from nonlinear systems theory. In turn, this enables architectural design decisions before experimentation. Sufficient conditions for global stability of the recurrent unit are obtained, motivating a novel scheme for constructing hidden-to-hidden matrices. Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks, including computer vision, language modeling and speech prediction tasks. Finally, through Hessian-based analysis we demonstrate that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (19)

• On the difficulty of training Recurrent Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• On orthogonality and learning recurrent networks with long term dependencies
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Full-Capacity Unitary Recurrent Neural Networks
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Stable Recurrent Models
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Lagrangian Neural Networks
• Recurrent Orthogonal Networks and Long-Memory Tasks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Trivializations for Gradient-Based Optimization on Manifolds
• PyHessian: Neural Networks Through the Lens of the Hessian
• IMEXnet: A Forward Stable Deep Neural Network
• Recurrent Neural Networks in the Eye of Differential Equations
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks

Cited by in corpus (10)

• Efficiently Modeling Long Sequences with Structured State Spaces
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Continuous-in-Depth Neural Networks
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• UnICORNN: A recurrent model for learning very long time dependencies
• Framing RNN as a kernel method: A neural ODE approach
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• Noisy Recurrent Neural Networks
• Sparse Flows: Pruning Continuous-depth Models
• Long Expressive Memory for Sequence Modeling