Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs

Abstract

We propose a simple interpolation-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method (known in the literature as "adjoint method") to train neural ODEs on classification, density estimation, and inference approximation tasks. We also propose a theoretical justification of our approach using logarithmic norm formalism. As a result, our method allows faster model training than the reverse dynamic method that was confirmed and validated by extensive numerical experiments for several standard benchmarks.

References in corpus (6)

• Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Multi-level Residual Networks from Dynamical Systems View
• ANODEV2: A Coupled Neural ODE Evolution Framework
• InfoCNF: An Efficient Conditional Continuous Normalizing Flow with Adaptive Solvers
• Towards Understanding Normalization in Neural ODEs

Cited by in corpus (3)

• Neural Closure Models for Dynamical Systems
• Neural Integral Equations
• Diffeomorphic Transformations for Time Series Analysis: An Efficient Approach to Nonlinear Warping