Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty

Abstract

The clinical interest is often to measure the volume of a structure, which is typically derived from a segmentation. In order to evaluate and compare segmentation methods, the similarity between a segmentation and a predefined ground truth is measured using popular discrete metrics, such as the Dice score. Recent segmentation methods use a differentiable surrogate metric, such as soft Dice, as part of the loss function during the learning phase. In this work, we first briefly describe how to derive volume estimates from a segmentation that is, potentially, inherently uncertain or ambiguous. This is followed by a theoretical analysis and an experimental validation linking the inherent uncertainty to common loss functions for training CNNs, namely cross-entropy and soft Dice. We find that, even though soft Dice optimization leads to an improved performance with respect to the Dice score and other measures, it may introduce a volume bias for tasks with high inherent uncertainty. These findings indicate some of the method's clinical limitations and suggest doing a closer ad-hoc volume analysis with an optional re-calibration step.

18 pages, 7 figures, 3 tables, published in Elsevier Medical Image Analysis (2021)

References in corpus (9)

• Fully Convolutional Networks for Semantic Segmentation
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• On Calibration of Modern Neural Networks
• Domain-adversarial neural networks to address the appearance variability of histopathology images
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index
• Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
• Cascaded V-Net using ROI masks for brain tumor segmentation

Cited by in corpus (1)

• The Dice loss in the context of missing or empty labels: Introducing $\Phi$ and $\epsilon$