QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results

Abstract

Deep learning (DL) models have provided state-of-the-art performance in various medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder translating DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties could enable clinical review of the most uncertain regions, thereby building trust and paving the way toward clinical translation. Several uncertainty estimation methods have recently been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentage of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, highlighting the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility, our evaluation code is made publicly available at: https://github.com/RagMeh11/QU-BraTS.

Accepted for publication at the Journal of Machine Learning for Biomedical Imaging (MELBA): https://www.melba-journal.org/papers/2022:026.html

References in corpus (17)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• On Calibration of Modern Neural Networks
• The Medical Segmentation Decathlon
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• Deep Bayesian Active Learning with Image Data
• Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challenge
• Common Limitations of Image Processing Metrics: A Picture Story
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• TuNet: End-to-end Hierarchical Brain Tumor Segmentation using Cascaded Networks
• DeepMRSeg: A convolutional deep neural network for anatomy and abnormality segmentation on MR images
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• HAD-Net: A Hierarchical Adversarial Knowledge Distillation Network for Improved Enhanced Tumour Segmentation Without Post-Contrast Images
• The Probabilistic Object Detection Challenge
• A Decoupled Uncertainty Model for MRI Segmentation Quality Estimation

Cited by in corpus (6)

• USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations
• Uncertainty Estimation for Heatmap-based Landmark Localization
• Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries
• Improving Robustness and Reliability in Medical Image Classification with Latent-Guided Diffusion and Nested-Ensembles
• Structural-Based Uncertainty in Deep Learning Across Anatomical Scales: Analysis in White Matter Lesion Segmentation
• Novel structural-scale uncertainty measures and error retention curves: application to multiple sclerosis