Augmented Neural ODEs

Abstract

We show that Neural Ordinary Differential Equations (ODEs) learn representations that preserve the topology of the input space and prove that this implies the existence of functions Neural ODEs cannot represent. To address these limitations, we introduce Augmented Neural ODEs which, in addition to being more expressive models, are empirically more stable, generalize better and have a lower computational cost than Neural ODEs.

NeurIPS camera ready, additional experiments, additional datasets, discussion on relation to other models

References in corpus (2)

• Invertible Residual Networks
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models

Cited by in corpus (30)

• Transformers for Modeling Physical Systems
• Scalable Gradients for Stochastic Differential Equations
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Graph Neural Ordinary Differential Equations
• Continuous Graph Neural Networks
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• On Robustness of Neural Ordinary Differential Equations
• Galaxy Morphology Classification using Neural Ordinary Differential Equations
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
• Robust learning with implicit residual networks
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Mean-Field Neural ODEs via Relaxed Optimal Control
• Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
• The Hintons in your Neural Network: a Quantum Field Theory View of Deep Learning
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• A General Framework for Survival Analysis and Multi-State Modelling
• Neural ODE with Temporal Convolution and Time Delay Neural Networks for Small-Footprint Keyword Spotting
• Neural ODEs for Image Segmentation with Level Sets
• Continuous-Depth Neural Models for Dynamic Graph Prediction
• Differential equations as models of deep neural networks
• Neural ODEs with stochastic vector field mixtures
• Latent Space Refinement for Deep Generative Models
• Optimization Induced Equilibrium Networks
• Characteristic Neural Ordinary Differential Equations
• Augmented KRnet for density estimation and approximation
• A unified framework for Hamiltonian deep neural networks
• OCT-GAN: Neural ODE-based Conditional Tabular GANs
• Transfer Learning using Neural Ordinary Differential Equations
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy
• STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction