Meta-Solver for Neural Ordinary Differential Equations
arXiv:2103.08561
Abstract
A conventional approach to train neural ordinary differential equations (ODEs) is to fix an ODE solver and then learn the neural network's weights to optimize a target loss function. However, such an approach is tailored for a specific discretization method and its properties, which may not be optimal for the selected application and yield the overfitting to the given solver. In our paper, we investigate how the variability in solvers' space can improve neural ODEs performance. We consider a family of Runge-Kutta methods that are parameterized by no more than two scalar variables. Based on the solvers' properties, we propose an approach to decrease neural ODEs overfitting to the pre-defined solver, along with a criterion to evaluate such behaviour. Moreover, we show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks. Recently it was shown that neural ODEs demonstrate superiority over conventional CNNs in terms of robustness. Our work demonstrates that the model robustness can be further improved by optimizing solver choice for a given task. The source code to reproduce our experiments is available at https://github.com/juliagusak/neural-ode-metasolver.
References in corpus (6)
- Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
- ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
- Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows
- Continuous-in-Depth Neural Networks
- Real-time Classification from Short Event-Camera Streams using Input-filtering Neural ODEs
- MRI Image Reconstruction via Learning Optimization Using Neural ODEs