On Stein Variational Neural Network Ensembles
arXiv:2106.10760
Abstract
Ensembles of deep neural networks have achieved great success recently, but they do not offer a proper Bayesian justification. Moreover, while they allow for averaging of predictions over several hypotheses, they do not provide any guarantees for their diversity, leading to redundant solutions in function space. In contrast, particle-based inference methods, such as Stein variational gradient descent (SVGD), offer a Bayesian framework, but rely on the choice of a kernel to measure the similarity between ensemble members. In this work, we study different SVGD methods operating in the weight space, function space, and in a hybrid setting. We compare the SVGD approaches to other ensembling-based methods in terms of their theoretical properties and assess their empirical performance on synthetic and real-world tasks. We find that SVGD using functional and hybrid kernels can overcome the limitations of deep ensembles. It improves on functional diversity and uncertainty estimation and approaches the true Bayesian posterior more closely. Moreover, we show that using stochastic SVGD updates, as opposed to the standard deterministic ones, can further improve the performance.
References in corpus (30)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Weight Uncertainty in Neural Networks
- Deep Ensembles: A Loss Landscape Perspective
- Bayesian Deep Learning and a Probabilistic Perspective of Generalization
- A Complete Recipe for Stochastic Gradient MCMC
- Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
- Functional Variational Bayesian Neural Networks
- Snapshot Ensembles: Train 1, get M for free
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Stein Variational Gradient Descent as Gradient Flow
- Message Passing Stein Variational Gradient Descent
- Stein Variational Gradient Descent With Matrix-Valued Kernels
- Understanding and Accelerating Particle-Based Variational Inference
- On the geometry of Stein variational gradient descent
- Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
- Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
- Uncertainty Quantification and Deep Ensembles
- Robust Out-of-distribution Detection for Neural Networks
- Stochastic Gradient MCMC with Repulsive Forces
- Bayesian Neural Network Priors Revisited
- The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
- Variational Tempering
- Understanding symmetries in deep networks
- BNNpriors: A library for Bayesian neural network inference with different prior distributions
- Understanding Variational Inference in Function-Space
- Annealed Stein Variational Gradient Descent
- Stochastic Stein Discrepancies
- Stein Self-Repulsive Dynamics: Benefits From Past Samples
- Kernel Stein Generative Modeling
- Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference