Uncertainty in Neural Networks: Approximately Bayesian Ensembling

Abstract

Understanding the uncertainty of a neural network's (NN) predictions is essential for many purposes. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to large numbers of parameters and data. Ensembling NNs provides an easily implementable, scalable method for uncertainty quantification, however, it has been criticised for not being Bayesian. This work proposes one modification to the usual process that we argue does result in approximate Bayesian inference; regularising parameters about values drawn from a distribution which can be set equal to the prior. A theoretical analysis of the procedure in a simplified setting suggests the recovered posterior is centred correctly but tends to have an underestimated marginal variance, and overestimated correlation. However, two conditions can lead to exact recovery. We argue that these conditions are partially present in NNs. Empirical evaluations demonstrate it has an advantage over standard ensembling, and is competitive with variational methods.

arXiv commentPlease cite as published in AISTATS 2020

References in corpus (7)

• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Understanding Measures of Uncertainty for Adversarial Example Detection
• Deep Exploration via Randomized Value Functions
• Polynomial Regression As an Alternative to Neural Nets
• Ensemble Sampling
• On the Importance of Strong Baselines in Bayesian Deep Learning
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions

Cited by in corpus (9)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Single Shot MC Dropout Approximation
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Rapid Risk Minimization with Bayesian Models Through Deep Learning Approximation
• Pathologies in priors and inference for Bayesian transformers