Stiff Neural Ordinary Differential Equations

Abstract

Neural Ordinary Differential Equations (ODE) are a promising approach to learn dynamic models from time-series data in science and engineering applications. This work aims at learning Neural ODE for stiff systems, which are usually raised from chemical kinetic modeling in chemical and biological systems. We first show the challenges of learning neural ODE in the classical stiff ODE systems of Robertson's problem and propose techniques to mitigate the challenges associated with scale separations in stiff systems. We then present successful demonstrations in stiff systems of Robertson's problem and an air pollution problem. The demonstrations show that the usage of deep networks with rectified activations, proper scaling of the network outputs as well as loss functions, and stabilized gradient calculations are the key techniques enabling the learning of stiff neural ODE. The success of learning stiff neural ODE opens up possibilities of using neural ODEs in applications with widely varying time-scales, like chemical dynamics in energy conversion, environmental engineering, and the life sciences.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• Stiff-PINN: Physics-Informed Neural Network for Stiff Chemical Kinetics
• Autonomous Discovery of Unknown Reaction Pathways from Data by Chemical Reaction Neural Network
• Time-series learning of latent-space dynamics for reduced-order model closure
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Accelerating Simulation of Stiff Nonlinear Systems using Continuous-Time Echo State Networks
• STEER: Simple Temporal Regularization For Neural ODEs

Cited by in corpus (15)

• KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics
• Residual-based physics-informed transfer learning: A hybrid method for accelerating long-term CFD simulations via deep learning
• Parsimonious Physics-Informed Random Projection Neural Networks for Initial-Value Problems of ODEs and index-1 DAEs
• Interpretable Polynomial Neural Ordinary Differential Equations
• Differentiable Programming for Earth System Modeling
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• Neural Ordinary Differential Equations for Model Order Reduction of Stiff Systems
• ChemKANs for Combustion Chemistry Modeling and Acceleration
• Abstract Interpretation of Fixpoint Iterators with Applications to Neural Networks
• Fredholm Neural Networks
• Forecasting with an N-dimensional Langevin Equation and a Neural-Ordinary Differential Equation
• NORi: An ML-Augmented Ocean Boundary Layer Parameterization
• Investigating the Surrogate Modeling Capabilities of Continuous Time Echo State Networks
• Neural Differential Appearance Equations
• Reinforcement learning-based adaptive time-integration for nonsmooth dynamics