Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning

Abstract

The use of ensembles of neural networks (NNs) for the quantification of predictive uncertainty is widespread. However, the current justification is intuitive rather than analytical. This work proposes one minor modification to the normal ensembling methodology, which we prove allows the ensemble to perform Bayesian inference, hence converging to the corresponding Gaussian Process as both the total number of NNs, and the size of each, tend to infinity. This working paper provides early-stage results in a reinforcement learning setting, analysing the practicality of the technique for an ensemble of small, finite number. Using the uncertainty estimates produced by anchored ensembles to govern the exploration-exploitation process results in steadier, more stable learning.

References in corpus (4)

• Overcoming catastrophic forgetting in neural networks
• Weight Uncertainty in Neural Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

Cited by in corpus (7)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Exploration in Deep Reinforcement Learning: A Survey
• Materials property prediction using symmetry-labeled graphs as atomic-position independent descriptors
• Empowering Bayesian Neural Networks with Functional Priors through Anchored Ensembling for Mechanics Surrogate Modeling Applications
• Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• Pathologies in priors and inference for Bayesian transformers