Direct Uncertainty Prediction for Medical Second Opinions
arXiv:1807.01771
Abstract
The issue of disagreements amongst human experts is a ubiquitous one in both machine learning and medicine. In medicine, this often corresponds to doctor disagreements on a patient diagnosis. In this work, we show that machine learning models can be trained to give uncertainty scores to data instances that might result in high expert disagreements. In particular, they can identify patient cases that would benefit most from a medical second opinion. Our central methodological finding is that Direct Uncertainty Prediction (DUP), training a model to predict an uncertainty score directly from the raw patient features, works better than Uncertainty Via Classification, the two-step process of training a classifier and postprocessing the output distribution to give an uncertainty score. We show this both with a theoretical result, and on extensive evaluations on a large scale medical imaging application.
arXiv commentAccepted for publication at ICML 2019
Cited by in corpus (13)
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
- Modality specific U-Net variants for biomedical image segmentation: A survey
- Reliability and Interpretability in Science and Deep Learning
- Consistent Estimators for Learning to Defer to an Expert
- Transfer Learning for Oral Cancer Detection using Microscopic Images
- Improving Medical Image Classification with Label Noise Using Dual-uncertainty Estimation
- Selective Classification Can Magnify Disparities Across Groups
- A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
- Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
- Evaluating AI systems under uncertain ground truth: a case study in dermatology
- Differentiable Learning Under Triage
- Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain
- Regression Under Human Assistance