Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit

Abstract

In deep latent Gaussian models, the latent variable is generated by a time-inhomogeneous Markov chain, where at each time step we pass the current state through a parametric nonlinear map, such as a feedforward neural net, and add a small independent Gaussian perturbation. This work considers the diffusion limit of such models, where the number of layers tends to infinity, while the step size and the noise variance tend to zero. The limiting latent object is an Itô diffusion process that solves a stochastic differential equation (SDE) whose drift and diffusion coefficient are implemented by neural nets. We develop a variational inference framework for these \textit{neural SDEs} via stochastic automatic differentiation in Wiener space, where the variational approximations to the posterior are obtained by Girsanov (mean-shift) transformation of the standard Wiener process and the computation of gradients is based on the theory of stochastic flows. This permits the use of black-box SDE solvers and automatic differentiation for end-to-end inference. Experimental results with synthetic data are provided.

References in corpus (11)

• Auto-Encoding Variational Bayes
• Solving high-dimensional partial differential equations using deep learning
• Convergence of the Deep BSDE Method for Coupled FBSDEs
• Neural Jump Stochastic Differential Equations
• DiffEqFlux.jl - A Julia Library for Neural Differential Equations
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Identity Matters in Deep Learning
• Continuous-Time Flows for Efficient Inference and Density Estimation
• Regularization under diffusion and anti-concentration of the information content
• Deep learning with differential Gaussian process flows
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems

Cited by in corpus (29)

• Normalizing Flows: An Introduction and Review of Current Methods
• Neural Controlled Differential Equations for Irregular Time Series
• Scalable Gradients for Stochastic Differential Equations
• Neural Jump Stochastic Differential Equations
• Dissecting Neural ODEs
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Maximum Likelihood Training of Score-Based Diffusion Models
• Stochastic Normalizing Flows
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Statistical Inference for Generative Models with Maximum Mean Discrepancy
• Learning the temporal evolution of multivariate densities via normalizing flows
• Bayesian Mechanics for Stationary Processes
• An end-to-end deep learning approach for extracting stochastic dynamical systems with $\alpha$-stable Lévy noise
• Stochastic Normalizing Flows
• Heavy Ball Neural Ordinary Differential Equations
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Differentiable Physics: A Position Piece
• Stochasticity in Neural ODEs: An Empirical Study
• Neural SDEs as Infinite-Dimensional GANs
• Noisy Recurrent Neural Networks
• Probabilistic Numeric Convolutional Neural Networks
• Learning Continuous-Time Dynamics by Stochastic Differential Networks
• Policy Analysis using Synthetic Controls in Continuous-Time
• Latent Neural Differential Equations for Video Generation
• Efficient and Accurate Gradients for Neural SDEs
• Meta-Solver for Neural Ordinary Differential Equations
• Deep limits and cut-off phenomena for neural networks
• Estimating Stochastic Poisson Intensities Using Deep Latent Models
• Neural Options Pricing