Neural Rough Differential Equations for Long Time Series

Abstract

Neural controlled differential equations (CDEs) are the continuous-time analogue of recurrent neural networks, as Neural ODEs are to residual networks, and offer a memory-efficient continuous-time way to model functions of potentially irregular time series. Existing methods for computing the forward pass of a Neural CDE involve embedding the incoming time series into path space, often via interpolation, and using evaluations of this path to drive the hidden state. Here, we use rough path theory to extend this formulation. Instead of directly embedding into path space, we instead represent the input signal over small time intervals through its \textit{log-signature}, which are statistics describing how the signal drives a CDE. This is the approach for solving \textit{rough differential equations} (RDEs), and correspondingly we describe our main contribution as the introduction of Neural RDEs. This extension has a purpose: by generalising the Neural CDE approach to a broader class of driving signals, we demonstrate particular advantages for tackling long time series. In this regime, we demonstrate efficacy on problems of length up to 17k observations and observe significant training speed-ups, improvements in model performance, and reduced memory requirements compared to existing approaches.

Published at ICML 2021

References in corpus (8)

• Latent ODEs for Irregularly-Sampled Time Series
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Full-Capacity Unitary Recurrent Neural Networks
• Rough paths, Signatures and the modelling of functions on streams
• Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU
• Calculation of Iterated-Integral Signatures and Log Signatures
• Pathwise approximation of SDEs by coupling piecewise abelian rough paths
• Areas of areas generate the shuffle algebra

Cited by in corpus (10)

• Learning Differential Equations that are Easy to Solve
• UnICORNN: A recurrent model for learning very long time dependencies
• A Generalised Signature Method for Multivariate Time Series Feature Extraction
• Neural Controlled Differential Equations for Online Prediction Tasks
• Framing RNN as a kernel method: A neural ODE approach
• Neural SDEs as Infinite-Dimensional GANs
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• SigFormer: Signature Transformers for Deep Hedging
• Learning the Dynamics of Sparsely Observed Interacting Systems
• Continuous Latent Process Flows