Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge

Abstract

Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. Automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their method on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge (https://wmh.isi.uu.nl/). Sixty T1+FLAIR images from three MR scanners were released with manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. Segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: (1) Dice similarity coefficient, (2) modified Hausdorff distance (95th percentile), (3) absolute log-transformed volume difference, (4) sensitivity for detecting individual lesions, and (5) F1-score for individual lesions. Additionally, methods were ranked on their inter-scanner robustness. Twenty participants submitted their method for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.

Accepted for publication in IEEE Transactions on Medical Imaging

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Rethinking Atrous Convolution for Semantic Image Segmentation
• On the Compactness, Efficiency, and Representation of 3D Convolutional Networks: Brain Parcellation as a Pretext Task
• Deep Retinal Image Understanding

Cited by in corpus (33)

• Domain Adaptation for Medical Image Analysis: A Survey
• Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index
• Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation
• Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
• Applications of Generative Adversarial Networks in Neuroimaging and Clinical Neuroscience
• CrossMoDA 2021 challenge: Benchmark of Cross-Modality Domain Adaptation techniques for Vestibular Schwannoma and Cochlea Segmentation
• Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets
• USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations
• Unsupervised Brain Anomaly Detection and Segmentation with Transformers
• Liver segmentation and metastases detection in MR images using convolutional neural networks
• Automated Claustrum Segmentation in Human Brain MRI Using Deep Learning
• DeepMRSeg: A convolutional deep neural network for anatomy and abnormality segmentation on MR images
• Challenging Current Semi-Supervised Anomaly Segmentation Methods for Brain MRI
• Guided Reconstruction with Conditioned Diffusion Models for Unsupervised Anomaly Detection in Brain MRIs
• Theoretical analysis and experimental validation of volume bias of soft Dice optimized segmentation maps in the context of inherent uncertainty
• 3D Brain and Heart Volume Generative Models: A Survey
• Feedback Graph Attention Convolutional Network for Medical Image Enhancement
• A Systematic Collection of Medical Image Datasets for Deep Learning
• Domain Adaptive Medical Image Segmentation via Adversarial Learning of Disease-Specific Spatial Patterns
• Detecting Scatteredly-Distributed, Small, andCritically Important Objects in 3D OncologyImaging via Decision Stratification
• Spatially Covariant Lesion Segmentation
• NeRD: Neural Representation of Distribution for Medical Image Segmentation
• Brain Atlas Guided Attention U-Net for White Matter Hyperintensity Segmentation
• Effect of latent space distribution on the segmentation of images with multiple annotations
• Multi-Texture GAN: Exploring the Multi-Scale Texture Translation for Brain MR Images
• Automated neuroradiological support systems for multiple cerebrovascular disease markers -- A systematic review and meta-analysis
• Robust White Matter Hyperintensity Segmentation on Unseen Domain
• Uncertainty quantification for White Matter Hyperintensity segmentation detects silent failures and improves automated Fazekas quantification
• Patient-specific fine-tuning of CNNs for follow-up lesion quantification
• When Weak Becomes Strong: Robust Quantification of White Matter Hyperintensities in Brain MRI scans
• Region-of-interest guided Supervoxel Inpainting for Self-supervision
• Attention-Based Segmentation of WMHs and Differentiation of Vascular vs. Demyelinating Lesions