Dissecting Neural ODEs

Abstract

Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still an open challenge, as most applications apply them as generic black-box modules. In this work we "open the box", further developing the continuous-depth formulation with the aim of clarifying the influence of several design choices on the underlying dynamics.

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Wide Residual Networks
• A Variational Perspective on Accelerated Methods in Optimization
• Hamiltonian Neural Networks
• Deep Equilibrium Models
• Scalable Gradients for Stochastic Differential Equations
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Neural Jump Stochastic Differential Equations
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Graph Neural Ordinary Differential Equations
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• On Robustness of Neural Ordinary Differential Equations
• Hamiltonian Descent Methods
• ANODEV2: A Coupled Neural ODE Evolution Framework
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• IMEXnet: A Forward Stable Deep Neural Network
• Hypersolvers: Toward Fast Continuous-Depth Models
• TorchDyn: A Neural Differential Equations Library
• State Space Representations of Deep Neural Networks
• Analysis on the Nonlinear Dynamics of Deep Neural Networks: Topological Entropy and Chaos

Cited by in corpus (34)

• Neural Controlled Differential Equations for Irregular Time Series
• A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacoustic
• Graph Neural Ordinary Differential Equations
• Learning Differential Equations that are Easy to Solve
• Continuous-in-Depth Neural Networks
• Neural integration for constitutive equations using small data
• Comparison of neural closure models for discretised PDEs
• Hypersolvers: Toward Fast Continuous-Depth Models
• STEER: Simple Temporal Regularization For Neural ODEs
• Learning Dynamic Alignment via Meta-filter for Few-shot Learning
• TorchDyn: A Neural Differential Equations Library
• Stable Neural Flows
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• Momentum Residual Neural Networks
• Neural Hybrid Automata: Learning Dynamics with Multiple Modes and Stochastic Transitions
• Neural SDEs as Infinite-Dimensional GANs
• The Hintons in your Neural Network: a Quantum Field Theory View of Deep Learning
• LT-OCF: Learnable-Time ODE-based Collaborative Filtering
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models
• A General Framework for Survival Analysis and Multi-State Modelling
• Prediction-based One-shot Dynamic Parking Pricing
• A Shooting Formulation of Deep Learning
• Sparse Flows: Pruning Continuous-depth Models
• Spline parameterization of neural network controls for deep learning
• Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows
• LightMove: A Lightweight Next-POI Recommendation for Taxicab Rooftop Advertising
• Continuous-Depth Neural Models for Dynamic Graph Prediction
• Rethinking ResNets: Improved Stacking Strategies With High Order Schemes
• Second-Order Neural ODE Optimizer
• Characteristic Neural Ordinary Differential Equations
• Differentiable Multiple Shooting Layers
• Convergent Graph Solvers
• Implicit regularization of deep residual networks towards neural ODEs