Towards calibrated and scalable uncertainty representations for neural networks

Abstract

For many applications it is critical to know the uncertainty of a neural network's predictions. While a variety of neural network parameter estimation methods have been proposed for uncertainty estimation, they have not been rigorously compared across uncertainty measures. We assess four of these parameter estimation methods to calibrate uncertainty estimation using four different uncertainty measures: entropy, mutual information, aleatoric uncertainty and epistemic uncertainty. We evaluate the calibration of these parameter estimation methods using expected calibration error. Additionally, we propose a novel method of neural network parameter estimation called RECAST, which combines cosine annealing with warm restarts with Stochastic Gradient Langevin Dynamics, capturing more diverse parameter distributions. When benchmarked against mutilated image data, we show that RECAST is well-calibrated and when combined with predictive entropy and epistemic uncertainty it offers the best calibrated measure of uncertainty when compared to recent methods.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019): 4th workshop on Bayesian Deep Learning, Vancouver, Canada

References in corpus (8)

• A Survey on Deep Learning in Medical Image Analysis
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Weight Uncertainty in Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Sampling-based Bayesian Inference with gradient uncertainty
• A Parsimonious Tour of Bayesian Model Uncertainty

Cited by in corpus (5)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• The Unreasonable Effectiveness of Deep Evidential Regression
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• MCU-Net: A framework towards uncertainty representations for decision support system patient referrals in healthcare contexts
• Multivariate Deep Evidential Regression