Continuous-in-Depth Neural Networks
arXiv:2008.02389
Abstract
Recent work has attempted to interpret residual networks (ResNets) as one step of a forward Euler discretization of an ordinary differential equation, focusing mainly on syntactic algebraic similarities between the two systems. Discrete dynamical integrators of continuous dynamical systems, however, have a much richer structure. We first show that ResNets fail to be meaningful dynamical integrators in this richer sense. We then demonstrate that neural network models can learn to represent continuous dynamical systems, with this richer structure and properties, by embedding them into higher-order numerical integration schemes, such as the Runge Kutta schemes. Based on these insights, we introduce ContinuousNet as a continuous-in-depth generalization of ResNet architectures. ContinuousNets exhibit an invariance to the particular computational graph manifestation. That is, the continuous-in-depth model can be evaluated with different discrete time step sizes, which changes the number of layers, and different numerical integration schemes, which changes the graph connectivity. We show that this can be used to develop an incremental-in-depth training scheme that improves model quality, while significantly decreasing training time. We also show that, once trained, the number of units in the computational graph can even be decreased, for faster inference with little-to-no accuracy drop.
References in corpus (33)
- Adam: A Method for Stochastic Optimization
- Wide Residual Networks
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- FractalNet: Ultra-Deep Neural Networks without Residuals
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- Deep Equilibrium Models
- Scalable Gradients for Stochastic Differential Equations
- FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
- How to train your neural ODE: the world of Jacobian and kinetic regularization
- The Shattered Gradients Problem: If resnets are the answer, then what is the question?
- Dissecting Neural ODEs
- Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
- ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
- Hamiltonian Graph Networks with ODE Integrators
- Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction
- Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
- Graph Neural Ordinary Differential Equations
- Lipschitz Recurrent Neural Networks
- Deep Learning Theory Review: An Optimal Control and Dynamical Systems Perspective
- Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
- On Robustness of Neural Ordinary Differential Equations
- Forecasting Sequential Data using Consistent Koopman Autoencoders
- Approximation Capabilities of Neural ODEs and Invertible Residual Networks
- Transport Analysis of Infinitely Deep Neural Network
- Towards Understanding the Importance of Shortcut Connections in Residual Networks
- SNODE: Spectral Discretization of Neural ODEs for System Identification
- Towards Robust ResNet: A Small Step but A Giant Leap
- Towards Robust and Stable Deep Learning Algorithms for Forward Backward Stochastic Differential Equations
- Stable Neural Flows
- Shadowing Properties of Optimization Algorithms
- Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
- Residual Networks as Nonlinear Systems: Stability Analysis using Linearization
Cited by in corpus (10)
- Denoising Diffusion Implicit Models
- Beltrami Flow and Neural Diffusion on Graphs
- MALI: A memory efficient and reverse accurate integrator for Neural ODEs
- Noisy Recurrent Neural Networks
- Inverse modified differential equations for discovery of dynamics
- Spline parameterization of neural network controls for deep learning
- Meta-Solver for Neural Ordinary Differential Equations
- A Differential Geometry Perspective on Orthogonal Recurrent Models
- Long Expressive Memory for Sequence Modeling
- Diffeomorphic Transformations for Time Series Analysis: An Efficient Approach to Nonlinear Warping