Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning

Abstract

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support-they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.

Cited by in corpus (11)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Uncertainty Estimation Using a Single Deep Deterministic Neural Network
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks
• Active Testing: Sample-Efficient Model Evaluation
• A statistical theory of cold posteriors in deep neural networks
• Bayesian Neural Networks: Essentials
• Posterior Meta-Replay for Continual Learning
• Efficient Action Recognition Using Confidence Distillation
• Deep covariate-learning: optimising information extraction from terrain texture for geostatistical modelling applications
• Know Where To Drop Your Weights: Towards Faster Uncertainty Estimation