Approximation Capabilities of Neural ODEs and Invertible Residual Networks

Abstract

Neural ODEs and i-ResNet are recently proposed methods for enforcing invertibility of residual neural models. Having a generic technique for constructing invertible models can open new avenues for advances in learning systems, but so far the question of whether Neural ODEs and i-ResNets can model any continuous invertible function remained unresolved. Here, we show that both of these models are limited in their approximation capabilities. We then prove that any homeomorphism on a $p$-dimensional Euclidean space can be approximated by a Neural ODE operating on a $2p$-dimensional Euclidean space, and a similar result for i-ResNets. We conclude by showing that capping a Neural ODE or an i-ResNet with a single linear layer is sufficient to turn the model into a universal approximator for non-invertible continuous functions.

References in corpus (7)

• Spectral Normalization for Generative Adversarial Networks
• Variational Inference with Normalizing Flows
• Residual Flows for Invertible Generative Modeling
• ResNet with one-neuron hidden layers is a Universal Approximator
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• ANODEV2: A Coupled Neural ODE Evolution Framework

Cited by in corpus (14)

• Normalizing Flows: An Introduction and Review of Current Methods
• Dissecting Neural ODEs
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations
• Coupling-based Invertible Neural Networks Are Universal Diffeomorphism Approximators
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Continuous-in-Depth Neural Networks
• Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory
• STEER: Simple Temporal Regularization For Neural ODEs
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Learning Stable Galerkin Models of Turbulence with Differentiable Programming
• Inverse modified differential equations for discovery of dynamics
• Neural Ordinary Differential Equations for Data-Driven Reduced Order Modeling of Environmental Hydrodynamics
• Universal Approximation of Residual Flows in Maximum Mean Discrepancy