The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning

Abstract

One of the main challenges of deep learning tools is their inability to capture model uncertainty. While Bayesian deep learning can be used to tackle the problem, Bayesian neural networks often require more time and computational power to train than deterministic networks. Our work explores whether fully Bayesian networks are needed to successfully capture model uncertainty. We vary the number and position of Bayesian layers in a network and compare their performance on active learning with the MNIST dataset. We found that we can fully capture the model uncertainty by using only a few Bayesian layers near the output of the network, combining the advantages of deterministic and Bayesian networks.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Weight Uncertainty in Neural Networks
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Deep Bayesian Active Learning with Image Data
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Noise Contrastive Priors for Functional Uncertainty
• A Survey on Bayesian Deep Learning

Cited by in corpus (8)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Deep Bayesian Active Learning, A Brief Survey on Recent Advances
• Deep Ensembles from a Bayesian Perspective
• Bayesian Neural Networks: Essentials
• Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation