Neural Ordinary Differential Equations

Abstract

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing flows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

arXiv commentv6: Corrected Fig. 3c function-evaluation counts (thanks to Vanni et al.); typo fixes

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• NICE: Non-linear Independent Components Estimation
• Adaptive Computation Time for Recurrent Neural Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems
• Sylvester Normalizing Flows for Variational Inference
• Improving Variational Auto-Encoders using Householder Flow
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• The Stan Math Library: Reverse-Mode Automatic Differentiation in C++
• Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier
• Probabilistic ODE Solvers with Runge-Kutta Means
• Variable Computation in Recurrent Neural Networks
• Treatment-Response Models for Counterfactual Reasoning with Continuous-time, Continuous-valued Interventions
• Black-box Variational Inference for Stochastic Differential Equations
• Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

Cited by in corpus (158)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Time Series Forecasting With Deep Learning: A Survey
• Spatial-Temporal Graph ODE Networks for Traffic Flow Forecasting
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Machine Learning and Deep Learning -- A review for Ecologists
• Hamiltonian Neural Networks
• Data Driven Governing Equations Approximation Using Deep Neural Networks
• A fast and accurate physics-informed neural network reduced order model with shallow masked autoencoder
• Transformers for Modeling Physical Systems
• Deep learning of dynamics and signal-noise decomposition with time-stepping constraints
• A deep learning enabler for non-intrusive reduced order modeling of fluid flows
• The Neural Particle Method -- An Updated Lagrangian Physics Informed Neural Network for Computational Fluid Dynamics
• Yao.jl: Extensible, Efficient Framework for Quantum Algorithm Design
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Time-series learning of latent-space dynamics for reduced-order model closure
• Machine Learning on Neutron and X-Ray Scattering
• On Learning Hamiltonian Systems from Data
• Why should we add early exits to neural networks?
• Assessment of unsteady flow predictions using hybrid deep learning based reduced order models
• Denoising Diffusion Implicit Models
• Deep Convolutional Networks as shallow Gaussian Processes
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• Enabling real-time multi-messenger astrophysics discoveries with deep learning
• Hidden Fluid Mechanics: A Navier-Stokes Informed Deep Learning Framework for Assimilating Flow Visualization Data
• Deep learning prediction of patient response time course from early data via neural-pharmacokinetic/pharmacodynamic modeling
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning
• Efficient Deep Learning Techniques for Multiphase Flow Simulation in Heterogeneous Porous Media
• A posteriori learning for quasi-geostrophic turbulence parametrization
• Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
• DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation
• A differentiable programming method for quantum control
• A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations
• Nonlinear proper orthogonal decomposition for convection-dominated flows
• Competitive Inner-Imaging Squeeze and Excitation for Residual Network
• On Interpretability of Artificial Neural Networks: A Survey
• Poly-Spline Finite Element Method
• Recipes for when Physics Fails: Recovering Robust Learning of Physics Informed Neural Networks
• Continuous Graph Neural Networks
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Reducing the complexity of chemical networks via interpretable autoencoders
• Learning Pair Potentials using Differentiable Simulations
• A comparative study of physics-informed neural network models for learning unknown dynamics and constitutive relations
• Interpretable Polynomial Neural Ordinary Differential Equations
• On the design space between molecular mechanics and machine learning force fields
• Galaxy Morphology Classification using Neural Ordinary Differential Equations
• A Tutorial on Deep Latent Variable Models of Natural Language
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Enforcing constraints for interpolation and extrapolation in Generative Adversarial Networks
• Deep convolutional neural networks for uncertainty propagation in random fields
• A literature survey of matrix methods for data science
• Graph-Guided Network for Irregularly Sampled Multivariate Time Series
• Transport Analysis of Infinitely Deep Neural Network
• Learning the temporal evolution of multivariate densities via normalizing flows
• Graph Element Networks: adaptive, structured computation and memory
• Confusion of Tagged Perturbations in Forward Automatic Differentiation of Higher-Order Functions
• Towards Optimally Weighted Physics-Informed Neural Networks in Ocean Modelling
• An end-to-end deep learning approach for extracting stochastic dynamical systems with $\alpha$-stable Lévy noise
• Neural Ordinary Differential Equations for Semantic Segmentation of Individual Colon Glands
• STEER: Simple Temporal Regularization For Neural ODEs
• LagNetViP: A Lagrangian Neural Network for Video Prediction
• Robust learning with implicit residual networks
• The Power Spherical distribution
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Neural ODE to model and prognose thermoacoustic instability
• Elvet -- a neural network-based differential equation and variational problem solver
• Towards Robust ResNet: A Small Step but A Giant Leap
• Hierarchical Associative Memory
• HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks
• Mean-field Langevin System, Optimal Control and Deep Neural Networks
• Structural Inference of Networked Dynamical Systems with Universal Differential Equations
• Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks
• Neural Controlled Differential Equations for Online Prediction Tasks
• Temporal Normalizing Flows
• Layer-Parallel Training of Deep Residual Neural Networks
• NeuralFMU: Towards Structural Integration of FMUs into Neural Networks
• Enforcing constraints for time series prediction in supervised, unsupervised and reinforcement learning
• Bayesian Sparse learning with preconditioned stochastic gradient MCMC and its applications
• WaveNODE: A Continuous Normalizing Flow for Speech Synthesis
• Geometrization of deep networks for the interpretability of deep learning systems
• On Scalable and Efficient Computation of Large Scale Optimal Transport
• Spacetime Neural Network for High Dimensional Quantum Dynamics
• Extended dynamic mode decomposition with dictionary learning using neural ordinary differential equations
• Learning Potentials of Quantum Systems using Deep Neural Networks
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• Machine learning-based vorticity evolution and superresolution of homogeneous isotropic turbulence using wavelet projection
• Path Integral Sampler: a stochastic control approach for sampling
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
• Learning Stable Galerkin Models of Turbulence with Differentiable Programming
• Variational Deep Learning for the Identification and Reconstruction of Chaotic and Stochastic Dynamical Systems from Noisy and Partial Observations
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• A Low-Cost Neural ODE with Depthwise Separable Convolution for Edge Domain Adaptation on FPGAs
• Learning dynamics on invariant measures using PDE-constrained optimization
• Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• FD-Net with Auxiliary Time Steps: Fast Prediction of PDEs using Hessian-Free Trust-Region Methods
• Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows
• Adaptive POD Galerkin technique for reservoir simulation and optimization
• An Empirical Study: Extensive Deep Temporal Point Process
• Learning Continuous-Time Dynamics by Stochastic Differential Networks
• Generative Design of a Gas Turbine Combustor Using Invertible Neural Networks
• Physics-aware Spatiotemporal Modules with Auxiliary Tasks for Meta-Learning
• Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
• A research framework for writing differentiable PDE discretizations in JAX
• Matrix Lie Maps and Neural Networks for Solving Differential Equations
• Monotonic Gaussian Process Flow
• Learning a Generator Model from Terminal Bus Data
• Constructing differential equations using only a scalar time-series about continuous time chaotic dynamics
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression
• Partial Differential Equations is All You Need for Generating Neural Architectures -- A Theory for Physical Artificial Intelligence Systems
• Deep learning and differential equations for modeling changes in individual-level latent dynamics between observation periods
• Deep learning of parameterized equations with applications to uncertainty quantification
• Characteristic Neural Ordinary Differential Equations
• Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers
• Meta-Solver for Neural Ordinary Differential Equations
• Non-intrusive inference reduced order model for fluids using linear multistep neural network
• Reconstructing a dynamical system and forecasting time series by self-consistent deep learning
• On Tuning Neural ODE for Stability, Consistency and Faster Convergence
• Augmenting Neural Differential Equations to Model Unknown Dynamical Systems with Incomplete State Information
• ChronoMID - Cross-Modal Neural Networks for 3-D Temporal Medical Imaging Data
• Quantum Deformed Neural Networks
• Ordinary Differential Equation-based CNN for Channel Extrapolation over RIS-assisted Communication
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Deep Learning-based Time-varying Channel Estimation for RIS Assisted Communication
• STRODE: Stochastic Boundary Ordinary Differential Equation
• ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
• Forecasting Black Sigatoka Infection Risks with Latent Neural ODEs
• When Machine Learning Meets Multiscale Modeling in Chemical Reactions
• Lie Transform--based Neural Networks for Dynamics Simulation and Learning
• A Differentiable Solver Approach to Operator Inference
• Quasi-Autoregressive Residual (QuAR) Flows
• OCT-GAN: Neural ODE-based Conditional Tabular GANs
• Controllability, Multiplexing, and Transfer Learning in Networks using Evolutionary Learning
• Augmented KRnet for density estimation and approximation
• Enhancing Human-Machine Teaming for Medical Prognosis Through Neural Ordinary Differential Equations (NODEs)
• Conformer-based End-to-end Speech Recognition With Rotary Position Embedding
• Particle Flow Bayes' Rule
• Neural Mode Jump Monte Carlo
• Neural Networks on Groups
• STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction
• Neural Differential Equations for Inverse Modeling in Model Combustors
• Progressive Growing of Neural ODEs
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics
• Physics-Augmented Learning: A New Paradigm Beyond Physics-Informed Learning
• Molecular Flow Monte Carlo
• HeunNet: Extending ResNet using Heun's Methods
• RBM-Flow and D-Flow: Invertible Flows with Discrete Energy Base Spaces
• Neural Architecture Search via Bregman Iterations
• GEM: Group Enhanced Model for Learning Dynamical Control Systems
• Solving Image PDEs with a Shallow Network
• EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss
• Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics
• Spatiotemporal Local Propagation
• Weakly-supervised learning on Schrodinger equation
• The Medium Amplitude Response of Nonlinear Maxwell-Oldroyd Type Models in Simple Shear