Loss-Calibrated Approximate Inference in Bayesian Neural Networks

Abstract

Current approaches in approximate inference for Bayesian neural networks minimise the Kullback-Leibler divergence to approximate the true posterior over the weights. However, this approximation is without knowledge of the final application, and therefore cannot guarantee optimal predictions for a given task. To make more suitable task-specific approximations, we introduce a new loss-calibrated evidence lower bound for Bayesian neural networks in the context of supervised learning, informed by Bayesian decision theory. By introducing a lower bound that depends on a utility function, we ensure that our approximation achieves higher utility than traditional methods for applications that have asymmetric utility functions. Furthermore, in using dropout inference, we highlight that our new objective is identical to that of standard dropout neural networks, with an additional utility-dependent penalty term. We demonstrate our new loss-calibrated model with an illustrative medical example and a restricted model capacity experiment, and highlight failure modes of the comparable weighted cross entropy approach. Lastly, we demonstrate the scalability of our method to real world applications with per-pixel semantic segmentation on an autonomous driving data set.

12 pages, 12 figures

References in corpus (1)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Cited by in corpus (17)

• Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• Galaxy Zoo: Probabilistic Morphology through Bayesian CNNs and Active Learning
• Parameters Estimation for the Cosmic Microwave Background with Bayesian Neural Networks
• Embedded deep learning in ophthalmology: Making ophthalmic imaging smarter
• Dual Bayesian ResNet: A Deep Learning Approach to Heart Murmur Detection
• Introducing an Explicit Symplectic Integration Scheme for Riemannian Manifold Hamiltonian Monte Carlo
• Using Sparse Gaussian Processes for Predicting Robust Inertial Confinement Fusion Implosion Yields
• Variational Bayesian Decision-making for Continuous Utilities
• Taming the Cross Entropy Loss
• URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference Methods for Deep Neural Networks
• Super-resolution of Time-series Labels for Bootstrapped Event Detection
• Targeted Active Learning for Bayesian Decision-Making
• HumBugDB: A Large-scale Acoustic Mosquito Dataset
• On Learning Prediction-Focused Mixtures
• Beyond Marginal Uncertainty: How Accurately can Bayesian Regression Models Estimate Posterior Predictive Correlations?
• WiSE-ALE: Wide Sample Estimator for Approximate Latent Embedding