SNODE: Spectral Discretization of Neural ODEs for System Identification

Abstract

This paper proposes the use of spectral element methods \citep{canuto_spectral_1988} for fast and accurate training of Neural Ordinary Differential Equations (ODE-Nets; \citealp{Chen2018NeuralOD}) for system identification. This is achieved by expressing their dynamics as a truncated series of Legendre polynomials. The series coefficients, as well as the network weights, are computed by minimizing the weighted sum of the loss function and the violation of the ODE-Net dynamics. The problem is solved by coordinate descent that alternately minimizes, with respect to the coefficients and the weights, two unconstrained sub-problems using standard backpropagation and gradient methods. The resulting optimization scheme is fully time-parallel and results in a low memory footprint. Experimental comparison to standard methods, such as backpropagation through explicit solvers and the adjoint technique \citep{Chen2018NeuralOD}, on training surrogate models of small and medium-scale dynamical systems shows that it is at least one order of magnitude faster at reaching a comparable value of the loss function. The corresponding testing MSE is one order of magnitude smaller as well, suggesting generalization capabilities increase.

arXiv commentPublished as a conference paper at ICLR 2020

References in corpus (10)

• LSTM: A Search Space Odyssey
• On the difficulty of training Recurrent Neural Networks
• Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations
• Using Machine Learning to Replicate Chaotic Attractors and Calculate Lyapunov Exponents from Data
• A Review of Pseudospectral Optimal Control: From Theory to Flight
• Highway and Residual Networks learn Unrolled Iterative Estimation
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Distributed optimization of deeply nested systems
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Physics Guided Recurrent Neural Networks For Modeling Dynamical Systems: Application to Monitoring Water Temperature And Quality In Lakes

Cited by in corpus (14)

• Neural Controlled Differential Equations for Irregular Time Series
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• Learning Long-Term Dependencies in Irregularly-Sampled Time Series
• Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems
• Continuous-in-Depth Neural Networks
• Causal Navigation by Continuous-time Neural Networks
• Heavy Ball Neural Ordinary Differential Equations
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs
• Inverse modified differential equations for discovery of dynamics
• A non-autonomous equation discovery method for time signal classification
• "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms
• Faster Policy Learning with Continuous-Time Gradients
• Neural Delay Differential Equations