Identifying nonlinear dynamical systems with multiple time scales and long-range dependencies
arXiv:1910.03471
Abstract
A main theoretical interest in biology and physics is to identify the nonlinear dynamical system (DS) that generated observed time series. Recurrent Neural Networks (RNNs) are, in principle, powerful enough to approximate any underlying DS, but in their vanilla form suffer from the exploding vs. vanishing gradients problem. Previous attempts to alleviate this problem resulted either in more complicated, mathematically less tractable RNN architectures, or strongly limited the dynamical expressiveness of the RNN. Here we address this issue by suggesting a simple regularization scheme for vanilla RNNs with ReLU activation which enables them to solve long-range dependency problems and express slow time scales, while retaining a simple mathematical structure which makes their DS properties partly analytically accessible. We prove two theorems that establish a tight connection between the regularized RNN dynamics and its gradients, illustrate on DS benchmarks that our regularization approach strongly eases the reconstruction of DS which harbor widely differing time scales, and show that our method is also en par with other long-range architectures like LSTMs on several tasks.
References in corpus (16)
- Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
- Data-driven discovery of coordinates and governing equations
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- On orthogonality and learning recurrent networks with long term dependencies
- Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI
- Hybrid modeling and prediction of dynamical systems
- Long-term Forecasting using Higher Order Tensor RNNs
- Improving performance of recurrent neural network with relu nonlinearity
- A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements
- Learning Dynamical Systems from Partial Observations
- Learning interpretable continuous-time models of latent stochastic dynamical systems
- State Space LSTM Models with Particle MCMC Inference
- Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
- Smoothing and parameter estimation by soft-adherence to governing equations
- Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems
- Transformation of ReLU-based recurrent neural networks from discrete-time to continuous-time