Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows

Abstract

We compare the discretize-optimize (Disc-Opt) and optimize-discretize (Opt-Disc) approaches for time-series regression and continuous normalizing flows (CNFs) using neural ODEs. Neural ODEs are ordinary differential equations (ODEs) with neural network components. Training a neural ODE is an optimal control problem where the weights are the controls and the hidden features are the states. Every training iteration involves solving an ODE forward and another backward in time, which can require large amounts of computation, time, and memory. Comparing the Opt-Disc and Disc-Opt approaches in image classification tasks, Gholami et al. (2019) suggest that Disc-Opt is preferable due to the guaranteed accuracy of gradients. In this paper, we extend the comparison to neural ODEs for time-series regression and CNFs. Unlike in classification, meaningful models in these tasks must also satisfy additional requirements beyond accurate final-time output, e.g., the invertibility of the CNF. Through our numerical experiments, we demonstrate that with careful numerical treatment, Disc-Opt methods can achieve similar performance as Opt-Disc at inference with drastically reduced training costs. Disc-Opt reduced costs in six out of seven separate problems with training time reduction ranging from 39% to 97%, and in one case, Disc-Opt reduced training from nine days to less than one day.

19 pages, 8 figures

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Variational Inference with Normalizing Flows
• Deep AutoRegressive Networks
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
• Transport Analysis of Infinitely Deep Neural Network

Cited by in corpus (10)

• Learning Differential Equations that are Easy to Solve
• A Neural Network Approach for High-Dimensional Optimal Control Applied to Multi-Agent Path Finding
• Multiple shooting for training neural differential equations on time series
• Learning Physical Models that Can Respect Conservation Laws
• A Neural Network Approach Applied to Multi-Agent Optimal Control
• DISCO-DJ I: a differentiable Einstein-Boltzmann solver for cosmology
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• Scientific Machine Learning of Flow Resistance Using Universal Shallow Water Equations with Differentiable Programming
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
• Meta-Solver for Neural Ordinary Differential Equations