STEER: Simple Temporal Regularization For Neural ODEs

Abstract

Training Neural Ordinary Differential Equations (ODEs) is often computationally expensive. Indeed, computing the forward pass of such models involves solving an ODE which can become arbitrarily complex during training. Recent works have shown that regularizing the dynamics of the ODE can partially alleviate this. In this paper we propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. Further, the technique is orthogonal to several other methods proposed to regularize the dynamics of ODEs and as such can be used in conjunction with them. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.

arXiv commentNeurips 2020

References in corpus (22)

• Density estimation using Real NVP
• Neural Ordinary Differential Equations
• Normalizing Flows for Probabilistic Modeling and Inference
• Scalable Gradients for Stochastic Differential Equations
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Neural Jump Stochastic Differential Equations
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Dissecting Neural ODEs
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Learning Differential Equations that are Easy to Solve
• Stochastic Normalizing Flows
• Continuous Graph Neural Networks
• On Robustness of Neural Ordinary Differential Equations
• Fifty Years of Stiffness
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
• Stochastic Normalizing Flows
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Exchangeable Neural ODE for Set Modeling
• Stochasticity in Neural ODEs: An Empirical Study
• Time Dependence in Non-Autonomous Neural ODEs
• Simple Video Generation using Neural ODEs

Cited by in corpus (6)

• Stiff Neural Ordinary Differential Equations
• LMT: Longitudinal Mixing Training, a Framework to Predict Disease Progression from a Single Image
• A non-autonomous equation discovery method for time signal classification
• Meta-Solver for Neural Ordinary Differential Equations
• Second-Order Neural ODE Optimizer
• STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction