Hypersolvers: Toward Fast Continuous-Depth Models

Abstract

The infinite-depth paradigm pioneered by Neural ODEs has launched a renaissance in the search for novel dynamical system-inspired deep learning primitives; however, their utilization in problems of non-trivial size has often proved impossible due to poor computational scalability. This work paves the way for scalable Neural ODEs with time-to-prediction comparable to traditional discrete networks. We introduce hypersolvers, neural networks designed to solve ODEs with low overhead and theoretical guarantees on accuracy. The synergistic combination of hypersolvers and Neural ODEs allows for cheap inference and unlocks a new frontier for practical application of continuous-depth models. Experimental evaluations on standard benchmarks, such as sampling for continuous normalizing flows, reveal consistent pareto efficiency over classical numerical methods.

References in corpus (25)

• Decoupled Weight Decay Regularization
• Solving high-dimensional partial differential equations using deep learning
• Normalizing Flows: An Introduction and Review of Current Methods
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Hamiltonian Neural Networks
• Learning to Simulate Complex Physics with Graph Networks
• Data Driven Governing Equations Approximation Using Deep Neural Networks
• Deep Equilibrium Models
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Artificial Neural Network Methods in Quantum Mechanics
• Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems
• DeepMoD: Deep learning for Model Discovery in noisy data
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Solving differential equations with neural networks: Applications to the calculation of cosmological phase transitions
• Neural Jump Stochastic Differential Equations
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Newton vs the machine: solving the chaotic three-body problem using deep neural networks
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Dissecting Neural ODEs
• Multi-level Residual Networks from Dynamical Systems View
• Lagrangian Neural Networks
• Graph Neural Ordinary Differential Equations
• Neural Networks Trained to Solve Differential Equations Learn General Representations
• Time-symmetric integration in astrophysics
• TorchDyn: A Neural Differential Equations Library

Cited by in corpus (9)

• Dissecting Neural ODEs
• Learning Differential Equations that are Easy to Solve
• Accelerating Neural ODEs Using Model Order Reduction
• TorchDyn: A Neural Differential Equations Library
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
• GoTube: Scalable Stochastic Verification of Continuous-Depth Models
• Learnable-Differentiable Finite Volume Solver for Accelerated Simulation of Flows
• Differentiable Multiple Shooting Layers
• Neural Fixed-Point Acceleration for Convex Optimization