ANODEV2: A Coupled Neural ODE Evolution Framework
arXiv:1906.04596
Abstract
It has been observed that residual networks can be viewed as the explicit Euler discretization of an Ordinary Differential Equation (ODE). This observation motivated the introduction of so-called Neural ODEs, which allow more general discretization schemes with adaptive time stepping. Here, we propose ANODEV2, which is an extension of this approach that also allows evolution of the neural network parameters, in a coupled ODE-based formulation. The Neural ODE method introduced earlier is in fact a special case of this new more general framework. We present the formulation of ANODEV2, derive optimality conditions, and implement a coupled reaction-diffusion-advection version of this framework in PyTorch. We present empirical results using several different configurations of ANODEV2, testing them on multiple models on CIFAR-10. We report results showing that this coupled ODE-based framework is indeed trainable, and that it achieves higher accuracy, as compared to the baseline models as well as the recently-proposed Neural ODE approach.
References in corpus (2)
Cited by in corpus (12)
- Dissecting Neural ODEs
- Stochastic Normalizing Flows
- Approximation Capabilities of Neural ODEs and Invertible Residual Networks
- Bounded nonlinear forecasts of partially observed geophysical systems with physics-constrained deep learning
- Momentum Residual Neural Networks
- Machine learning structure preserving brackets for forecasting irreversible processes
- Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs
- MRI Image Reconstruction via Learning Optimization Using Neural ODEs
- Towards Understanding Normalization in Neural ODEs
- ODE-based Deep Network for MRI Reconstruction
- Learning Runge-Kutta Integration Schemes for ODE Simulation and Identification
- Transfer Learning using Neural Ordinary Differential Equations