Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index

Abstract

In many medical imaging and classical computer vision tasks, the Dice score and Jaccard index are used to evaluate the segmentation performance. Despite the existence and great empirical success of metric-sensitive losses, i.e. relaxations of these metrics such as soft Dice, soft Jaccard and Lovasz-Softmax, many researchers still use per-pixel losses, such as (weighted) cross-entropy to train CNNs for segmentation. Therefore, the target metric is in many cases not directly optimized. We investigate from a theoretical perspective, the relation within the group of metric-sensitive loss functions and question the existence of an optimal weighting scheme for weighted cross-entropy to optimize the Dice score and Jaccard index at test time. We find that the Dice score and Jaccard index approximate each other relatively and absolutely, but we find no such approximation for a weighted Hamming similarity. For the Tversky loss, the approximation gets monotonically worse when deviating from the trivial weight setting where soft Tversky equals soft Dice. We verify these results empirically in an extensive validation on six medical segmentation tasks and can confirm that metric-sensitive losses are superior to cross-entropy based loss functions in case of evaluation with Dice Score or Jaccard Index. This further holds in a multi-class setting, and across different object sizes and foreground/background ratios. These results encourage a wider adoption of metric-sensitive loss functions for medical segmentation tasks where the performance measure of interest is the Dice score or Jaccard index.

15 pages, 14 figures, accepted for publication in IEEE Transactions on Medical Imaging (2020)

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Fully Convolutional Networks for Semantic Segmentation
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
• Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice
• A Convex Surrogate Operator for General Non-Modular Loss Functions
• No New-Net

Cited by in corpus (18)

• Medical Image Segmentation Using Deep Learning: A Survey
• Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses
• DGNet: Distribution Guided Efficient Learning for Oil Spill Image Segmentation
• Distribution-aware Margin Calibration for Semantic Segmentation in Images
• blob loss: instance imbalance aware loss functions for semantic segmentation
• Deep Learning Analysis of Cardiac MRI in Legacy Datasets: Multi-Ethnic Study of Atherosclerosis
• Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty
• EllSeg-Gen, towards Domain Generalization for head-mounted eyetracking
• Evaluation of Data Augmentation and Loss Functions in Semantic Image Segmentation for Drilling Tool Wear Detection
• The Dice loss in the context of missing or empty labels: Introducing $\Phi$ and $\epsilon$
• FESS Loss: Feature-Enhanced Spatial Segmentation Loss for Optimizing Medical Image Analysis
• A Review of Benchmarks for Visual Defect Detection in the Manufacturing Industry
• Average Jaccard Index of Random Graphs
• Deep Structure Learning using Feature Extraction in Trained Projection Space
• PL-Net: Progressive Learning Network for Medical Image Segmentation
• Distribution-aware Margin Calibration for Medical Image Segmentation
• A new smart-cropping pipeline for prostate segmentation using deep learning networks
• Automated risk classification of colon biopsies based on semantic segmentation of histopathology images