Revisiting the Evaluation of Uncertainty Estimation and Its Application to Explore Model Complexity-Uncertainty Trade-Off

Abstract

Accurately estimating uncertainties in neural network predictions is of great importance in building trusted DNNs-based models, and there is an increasing interest in providing accurate uncertainty estimation on many tasks, such as security cameras and autonomous driving vehicles. In this paper, we focus on the two main use cases of uncertainty estimation, i.e. selective prediction and confidence calibration. We first reveal potential issues of commonly used quality metrics for uncertainty estimation in both use cases, and propose our new metrics to mitigate them. We then apply these new metrics to explore the trade-off between model complexity and uncertainty estimation quality, a critically missing work in the literature. Our empirical experiment results validate the superiority of the proposed metrics, and some interesting trends about the complexity-uncertainty trade-off are observed.

arXiv commentCVPR 2020 - Fair, Data Efficient and Trusted Computer Vision Workshop

References in corpus (18)

• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• What do we need to build explainable AI systems for the medical domain?
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Measuring Calibration in Deep Learning
• Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples
• Evaluating model calibration in classification
• Reducing Network Agnostophobia
• Distance-based Confidence Score for Neural Network Classifiers
• Leveraging Uncertainty Estimates for Predicting Segmentation Quality
• Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI
• Maximum Likelihood with Bias-Corrected Calibration is Hard-To-Beat at Label Shift Adaptation
• Confident Multiple Choice Learning
• Confidence Scoring Using Whitebox Meta-models with Linear Classifier Probes

Cited by in corpus (2)

• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Classification Confidence Estimation with Test-Time Data-Augmentation