Deep learning of dynamics and signal-noise decomposition with time-stepping constraints

Abstract

A critical challenge in the data-driven modeling of dynamical systems is producing methods robust to measurement error, particularly when data is limited. Many leading methods either rely on denoising prior to learning or on access to large volumes of data to average over the effect of noise. We propose a novel paradigm for data-driven modeling that simultaneously learns the dynamics and estimates the measurement noise at each observation. By constraining our learning algorithm, our method explicitly accounts for measurement error in the map between observations, treating both the measurement error and the dynamics as unknowns to be identified, rather than assuming idealized noiseless trajectories. We model the unknown vector field using a deep neural network, imposing a Runge-Kutta integrator structure to isolate this vector field, even when the data has a non-uniform timestep, thus constraining and focusing the modeling effort. We demonstrate the ability of this framework to form predictive models on a variety of canonical test problems of increasing complexity and show that it is robust to substantial amounts of measurement error. We also discuss issues with the generalizability of neural network models for dynamical systems and provide open-source code for all examples.

Updated a reference to reflect publication

References in corpus (8)

• On the difficulty of training Recurrent Neural Networks
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• The Loss Surfaces of Multilayer Networks
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Data Driven Governing Equations Approximation Using Deep Neural Networks
• Learning Koopman Invariant Subspaces for Dynamic Mode Decomposition
• Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems

Cited by in corpus (48)

• Data-driven discovery of coordinates and governing equations
• Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks
• Data Driven Governing Equations Approximation Using Deep Neural Networks
• Weak SINDy For Partial Differential Equations
• Data-Driven Deep Learning of Partial Differential Equations in Modal Space
• Weak SINDy: Galerkin-Based Data-Driven Model Selection
• Data-driven prediction of a multi-scale Lorenz 96 chaotic system using deep learning methods: Reservoir computing, ANN, and RNN-LSTM
• Construction of Reduced Order Models for Fluid Flows Using Deep Feedforward Neural Networks
• On the Structure of Time-delay Embedding in Linear Models of Non-linear Dynamical Systems
• Discovery of Nonlinear Dynamical Systems using a Runge-Kutta Inspired Dictionary-based Sparse Regression Approach
• On spike-and-slab priors for Bayesian equation discovery of nonlinear dynamical systems via sparse linear regression
• On generalized residue network for deep learning of unknown dynamical systems
• Neural Galerkin Schemes with Active Learning for High-Dimensional Evolution Equations
• Learning interaction kernels in heterogeneous systems of agents from multiple trajectories
• Extracting Interpretable Physical Parameters from Spatiotemporal Systems using Unsupervised Learning
• Sparse Identification of Truncation Errors
• An integrated data-driven computational pipeline with model order reduction for industrial and applied mathematics
• Extended Dynamic Mode Decomposition for Inhomogeneous Problems
• On the comparison of LES data-driven reduced order approaches for hydroacoustic analysis
• A Levenberg-Marquardt algorithm for sparse identification of dynamical systems
• Interpretable Conservation Law Estimation by Deriving the Symmetries of Dynamics from Trained Deep Neural Networks
• Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps
• Smoothing and parameter estimation by soft-adherence to governing equations
• Deep Euler method: solving ODEs by approximating the local truncation error of the Euler method
• Multiresolution Convolutional Autoencoders
• Robust Modeling of Unknown Dynamical Systems via Ensemble Averaged Learning
• Neural Dynamical Systems: Balancing Structure and Flexibility in Physical Prediction
• Discovery of Dynamics Using Linear Multistep Methods
• SubTSBR to tackle high noise and outliers for data-driven discovery of differential equations
• CINDy: Conditional gradient-based Identification of Non-linear Dynamics -- Noise-robust recovery
• Sparsely constrained neural networks for model discovery of PDEs
• Learning Dynamics from Noisy Measurements using Deep Learning with a Runge-Kutta Constraint
• LQResNet: A Deep Neural Network Architecture for Learning Dynamic Processes
• Hierarchical deep learning-based adaptive time-stepping scheme for multiscale simulations
• Data-driven learning of non-autonomous systems
• Deep Neural Network Modeling of Unknown Partial Differential Equations in Nodal Space
• Constrained or Unconstrained? Neural-Network-Based Equation Discovery from Data
• STENCIL-NET: Data-driven solution-adaptive discretization of partial differential equations
• Uncovering Closed-form Governing Equations of Nonlinear Dynamics from Videos
• Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations
• Methods to Recover Unknown Processes in Partial Differential Equations Using Data
• Learning Interaction Kernels for Agent Systems on Riemannian Manifolds
• Real-time simulation of parameter-dependent fluid flows through deep learning-based reduced order models
• Deep learning of parameterized equations with applications to uncertainty quantification
• Model discovery in the sparse sampling regime
• Variational inference formulation for a model-free simulation of a dynamical system with unknown parameters by a recurrent neural network
• Learning Low-Dimensional Quadratic-Embeddings of High-Fidelity Nonlinear Dynamics using Deep Learning
• Learning Dynamical Systems from Noisy Sensor Measurements using Multiple Shooting