Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
arXiv:2102.06559
Abstract
We perform scalable approximate inference in continuous-depth Bayesian neural networks. In this model class, uncertainty about separate weights in each layer gives hidden units that follow a stochastic differential equation. We demonstrate gradient-based stochastic variational inference in this infinite-parameter setting, producing arbitrarily-flexible approximate posteriors. We also derive a novel gradient estimator that approaches zero variance as the approximate posterior over weights approaches the true posterior. This approach brings continuous-depth Bayesian neural nets to a competitive comparison against discrete-depth alternatives, while inheriting the memory-efficient training and tunable precision of Neural ODEs.
References in corpus (11)
- On Calibration of Modern Neural Networks
- Score-Based Generative Modeling through Stochastic Differential Equations
- Weight Uncertainty in Neural Networks
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
- Latent ODEs for Irregularly-Sampled Time Series
- Fixup Initialization: Residual Learning Without Normalization
- Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
- Principled Weight Initialization for Hypernetworks
- Bayesian Neural Ordinary Differential Equations
- Theoretical guarantees for sampling and inference in generative models with latent diffusions
- Differential Bayesian Neural Nets