Neural Controlled Differential Equations for Irregular Time Series

Abstract

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

arXiv commentAccepted at NeurIPS 2020 (Spotlight)

References in corpus (24)

• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• BRITS: Bidirectional Recurrent Imputation for Time Series
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• Neural Jump Stochastic Differential Equations
• Dissecting Neural ODEs
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier
• Rough paths, Signatures and the modelling of functions on streams
• A generative adversarial network approach to calibration of local stochastic volatility models
• Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification
• Signature moments to characterize laws of stochastic processes
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Interpolation-Prediction Networks for Irregularly Sampled Time Series
• Learning stochastic differential equations using RNN with log signature features
• Discrete Event, Continuous Time RNNs
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• Dimension-free Euler estimates of rough differential equations
• Derivatives pricing using signature payoffs
• Stable Neural Flows
• Embedding and learning with signatures
• Stochasticity in Neural ODEs: An Empirical Study
• Neural ODEs with stochastic vector field mixtures

Cited by in corpus (13)

• Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs
• A Survey on Kolmogorov-Arnold Network
• Deep learning prediction of patient response time course from early data via neural-pharmacokinetic/pharmacodynamic modeling
• Neural Ordinary Differential Equation Control of Dynamics on Graphs
• Addressing Prediction Delays in Time Series Forecasting: A Continuous GRU Approach with Derivative Regularization
• Monotonic Neural Ordinary Differential Equation: Time-series Forecasting for Cumulative Data
• COPER: Continuous Patient State Perceiver
• Framing RNN as a kernel method: A neural ODE approach
• A non-autonomous equation discovery method for time signal classification
• Deep learning and differential equations for modeling changes in individual-level latent dynamics between observation periods
• Predicting the Need for Blood Transfusion in Intensive Care Units with Reinforcement Learning
• Learning the Dynamics of Sparsely Observed Interacting Systems
• sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging