Learning Long-Term Dependencies in Irregularly-Sampled Time Series

Abstract

Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. These models, however, face difficulties when the input data possess long-term dependencies. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. This phenomenon is expressed by the ordinary differential equation (ODE) representation of the hidden state, regardless of the ODE solver's choice. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state. This way, we encode a continuous-time dynamical flow within the RNN, allowing it to respond to inputs arriving at arbitrary time-lags while ensuring a constant error propagation through the memory path. We call these RNN models ODE-LSTMs. We experimentally show that ODE-LSTMs outperform advanced RNN-based counterparts on non-uniformly sampled data with long-term dependencies. All code and data is available at https://github.com/mlech26l/ode-lstms.

References in corpus (17)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• LSTM: A Search Space Odyssey
• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• On the difficulty of training Recurrent Neural Networks
• BRITS: Bidirectional Recurrent Imputation for Time Series
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• Neural Jump Stochastic Differential Equations
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification
• Learning to Control PDEs with Differentiable Physics
• Functional Autoregression for Sparsely Sampled Data
• Discrete Event, Continuous Time RNNs
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• State-Regularized Recurrent Neural Networks
• Recurrent Kernel Networks
• Scalable Linear Causal Inference for Irregularly Sampled Time Series with Long Range Dependencies

Cited by in corpus (7)

• STDEN: Towards Physics-Guided Neural Networks for Traffic Flow Prediction
• Optimized Three Deep Learning Models Based-PSO Hyperparameters for Beijing PM2.5 Prediction
• DeepUKF-VIN: Adaptively-tuned Deep Unscented Kalman Filter for 3D Visual-Inertial Navigation based on IMU-Vision-Net
• Neural Chronos ODE: Unveiling Temporal Patterns and Forecasting Future and Past Trends in Time Series Data
• Longitudinal Self-supervised Learning Using Neural Ordinary Differential Equation
• Modeling Irregular Astronomical Time Series with Neural Stochastic Delay Differential Equations
• TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification