Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations

Abstract

We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.

References in corpus (27)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• On Calibration of Modern Neural Networks
• Score-Based Generative Modeling through Stochastic Differential Equations
• Weight Uncertainty in Neural Networks
• Density estimation using Real NVP
• Neural Ordinary Differential Equations
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Bayesian Dark Knowledge
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• Latent ODEs for Irregularly-Sampled Time Series
• Scalable Gradients for Stochastic Differential Equations
• Practical Deep Learning with Bayesian Principles
• Fixup Initialization: Residual Learning Without Normalization
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
• Augmented Neural ODEs
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• Principled Weight Initialization for Hypernetworks
• Adversarial Distillation of Bayesian Neural Network Posteriors
• Approximation Capabilities of Neural ODEs and Invertible Residual Networks
• Black-box Variational Inference for Stochastic Differential Equations
• Bayesian Neural Ordinary Differential Equations
• Bayesian Hypernetworks
• Theoretical guarantees for sampling and inference in generative models with latent diffusions
• Stochasticity in Neural ODEs: An Empirical Study
• Differential Bayesian Neural Nets

Cited by in corpus (3)

• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• GoTube: Scalable Stochastic Verification of Continuous-Depth Models
• Efficient and Accurate Gradients for Neural SDEs