Learning Differential Equations that are Easy to Solve

Abstract

Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.

References in corpus (17)

• Sensitivity and Generalization in Neural Networks: an Empirical Study
• A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problems
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• Reproducing kernel Hilbert spaces of Gaussian priors
• Scalable Gradients for Stochastic Differential Equations
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• Dissecting Neural ODEs
• Multi-level Residual Networks from Dynamical Systems View
• On Robustness of Neural Ordinary Differential Equations
• Differentiable Molecular Simulations for Control and Learning
• Maximum a Posteriori State Path Estimation: Discretization Limits and their Interpretation
• Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows
• Neural Rough Differential Equations for Long Time Series
• Hypersolvers: Toward Fast Continuous-Depth Models
• Neural Ordinary Differential Equations for Semantic Segmentation of Individual Colon Glands
• Interpolation between Residual and Non-Residual Networks

Cited by in corpus (15)

• Deep learning of thermodynamics-aware reduced-order models from data
• Accelerating Neural ODEs Using Model Order Reduction
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• STEER: Simple Temporal Regularization For Neural ODEs
• Robust learning with implicit residual networks
• $m^{\ast }$ of two-dimensional electron gas: a neural canonical transformation study
• Framing RNN as a kernel method: A neural ODE approach
• Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling
• ResNet After All? Neural ODEs and Their Numerical Solution
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
• "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms
• Second-Order Neural ODE Optimizer
• Meta-Solver for Neural Ordinary Differential Equations
• Incorporating NODE with Pre-trained Neural Differential Operator for Learning Dynamics