Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain

Abstract

We propose an evaluation framework for class probability estimates (CPEs) in the presence of label uncertainty, which is commonly observed as diagnosis disagreement between experts in the medical domain. We also formalize evaluation metrics for higher-order statistics, including inter-rater disagreement, to assess predictions on label uncertainty. Moreover, we propose a novel post-hoc method called $alpha$-calibration, that equips neural network classifiers with calibrated distributions over CPEs. Using synthetic experiments and a large-scale medical imaging application, we show that our approach significantly enhances the reliability of uncertainty estimates: disagreement probabilities and posterior CPEs.

arXiv comment31 pages, 6 figures

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• On Calibration of Modern Neural Networks
• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• Evidential Deep Learning to Quantify Classification Uncertainty
• Deep Label Distribution Learning with Label Ambiguity
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Evaluating model calibration in classification
• Direct Uncertainty Prediction for Medical Second Opinions
• Calibration tests in multi-class classification: A unifying framework
• Being Bayesian about Categorical Probability
• TIMELY: Improving Labeling Consistency in Medical Imaging for Cell Type Classification