Exploiting Uncertainties from Ensemble Learners to Improve Decision-Making in Healthcare AI

Abstract

Ensemble learning is widely applied in Machine Learning (ML) to improve model performance and to mitigate decision risks. In this approach, predictions from a diverse set of learners are combined to obtain a joint decision. Recently, various methods have been explored in literature for estimating decision uncertainties using ensemble learning; however, determining which metrics are a better fit for certain decision-making applications remains a challenging task. In this paper, we study the following key research question in the selection of uncertainty metrics: when does an uncertainty metric outperforms another? We answer this question via a rigorous analysis of two commonly used uncertainty metrics in ensemble learning, namely ensemble mean and ensemble variance. We show that, under mild assumptions on the ensemble learners, ensemble mean is preferable with respect to ensemble variance as an uncertainty metric for decision making. We empirically validate our assumptions and theoretical results via an extensive case study: the diagnosis of referable diabetic retinopathy.

Preprint of submission to NeurIPS 2020

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• Deep Ensembles: A Loss Landscape Perspective
• Automatic Brain Tumor Segmentation using Convolutional Neural Networks with Test-Time Augmentation
• A One-Class Support Vector Machine Calibration Method for Time Series Change Point Detection
• An Encoder-Decoder Based Approach for Anomaly Detection with Application in Additive Manufacturing
• Augmenting Monte Carlo Dropout Classification Models with Unsupervised Learning Tasks for Detecting and Diagnosing Out-of-Distribution Faults
• Are Ensemble Classifiers Powerful Enough for the Detection and Diagnosis of Intermediate-Severity Faults?

Cited by in corpus (2)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Generalizing Fault Detection Against Domain Shifts Using Stratification-Aware Cross-Validation