Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results

Abstract

Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models.

This challenge was hosted in MICCAI 2024

References in corpus (15)

• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Decoupled Weight Decay Regularization
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• On Calibration of Modern Neural Networks
• TotalSegmentator: robust segmentation of 104 anatomical structures in CT images
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
• Classifier Calibration: A survey on how to assess and improve predicted class probabilities
• WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
• Vision-LSTM: xLSTM as Generic Vision Backbone
• AbdomenAtlas-8K: Annotating 8,000 CT Volumes for Multi-Organ Segmentation in Three Weeks
• Failure Detection in Medical Image Classification: A Reality Check and Benchmarking Testbed
• QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
• Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation