Recurrent Neural Networks in the Eye of Differential Equations

Abstract

To understand the fundamental trade-offs between training stability, temporal dynamics and architectural complexity of recurrent neural networks~(RNNs), we directly analyze RNN architectures using numerical methods of ordinary differential equations~(ODEs). We define a general family of RNNs--the ODERNNs--by relating the composition rules of RNNs to integration methods of ODEs at discrete time steps. We show that the degree of RNN's functional nonlinearity $n$ and the range of its temporal memory $t$ can be mapped to the corresponding stage of Runge-Kutta recursion and the order of time-derivative of the ODEs. We prove that popular RNN architectures, such as LSTM and URNN, fit into different orders of $n$-$t$-ODERNNs. This exact correspondence between RNN and ODE helps us to establish the sufficient conditions for RNN training stability and facilitates more flexible top-down designs of new RNN architectures using large varieties of toolboxes from numerical integration of ODEs. We provide such an example: Quantum-inspired Universal computing Neural Network~(QUNN), which reduces the required number of training parameters from polynomial in both data length and temporal memory length to only linear in temporal memory length.

arXiv comment25pages, 3 figures

References in corpus (1)

• Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

Cited by in corpus (11)

• Machine Learning on Neutron and X-Ray Scattering
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
• Symplectic Recurrent Neural Networks
• Lipschitz Recurrent Neural Networks
• Fast and Accurate Modeling of Transient-State Gradient-Spoiled Sequences by Recurrent Neural Networks
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• RNNs Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?
• Differential equations as models of deep neural networks
• RNN Training along Locally Optimal Trajectories via Frank-Wolfe Algorithm