Structured Stochastic Gradient MCMC
arXiv:2107.09028
Abstract
Stochastic gradient Markov Chain Monte Carlo (SGMCMC) is considered the gold standard for Bayesian inference in large-scale models, such as Bayesian neural networks. Since practitioners face speed versus accuracy tradeoffs in these models, variational inference (VI) is often the preferable option. Unfortunately, VI makes strong assumptions on both the factorization and functional form of the posterior. In this work, we propose a new non-parametric variational approximation that makes no assumptions about the approximate posterior's functional form and allows practitioners to specify the exact dependencies the algorithm should respect or break. The approach relies on a new Langevin-type algorithm that operates on a modified energy function, where parts of the latent variables are averaged over samples from earlier iterations of the Markov chain. This way, statistical dependencies can be broken in a controlled way, allowing the chain to mix faster. This scheme can be further modified in a "dropout" manner, leading to even more scalability. We test our scheme for ResNet-20 on CIFAR-10, SVHN, and FMNIST. In all cases, we find improvements in convergence speed and/or final accuracy compared to SG-MCMC and VI.
arXiv commentpaper accepted in ICML2022. Code can be found here https://github.com/ajboyd2/pytorch_lvi
References in corpus (15)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- MCMC using Hamiltonian dynamics
- On Calibration of Modern Neural Networks
- Variational Inference with Normalizing Flows
- Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
- Stochastic Gradient Hamiltonian Monte Carlo
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
- What Are Bayesian Neural Network Posteriors Really Like?
- Advances in Variational Inference
- Dropout as a Structured Shrinkage Prior
- Stochastic Gradient MCMC Methods for Hidden Markov Models