Metrics reloaded: Recommendations for image analysis validation

Abstract

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.

Shared first authors: Lena Maier-Hein, Annika Reinke. arXiv admin note: substantial text overlap with arXiv:2104.05642 Published in Nature Methods

References in corpus (13)

• Natural Scales in Geographical Patterns
• Shortcut Learning in Deep Neural Networks
• The Medical Segmentation Decathlon
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Understanding metric-related pitfalls in image analysis validation
• blob loss: instance imbalance aware loss functions for semantic segmentation
• A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification
• Analysis and Comparison of Classification Metrics
• Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient
• Beyond calibration: estimating the grouping loss of modern neural networks
• Sources of performance variability in deep learning-based polyp detection

Cited by in corpus (33)

• Segment Anything in Medical Images
• Understanding metric-related pitfalls in image analysis validation
• The Multi-modality Cell Segmentation Challenge: Towards Universal Solutions
• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• TotalSegmentator MRI: Robust Sequence-independent Segmentation of Multiple Anatomic Structures in MRI
• Similarity and Quality Metrics for MR Image-To-Image Translation
• Tracking and Mapping in Medical Computer Vision: A Review
• Brain tumour segmentation with incomplete imaging data
• A study of why we need to reassess full reference image quality assessment with medical images
• A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge
• AI-powered skin spectral imaging enables instant sepsis diagnosis and outcome prediction in critically ill patients
• Defining the boundaries: challenges and advances in identifying cells in microscopy images
• Non-inferiority of Deep Learning Acute Ischemic Stroke Segmentation on Non-Contrast CT Compared to Expert Neuroradiologists
• Random Expert Sampling for Deep Learning Segmentation of Acute Ischemic Stroke on Non-contrast CT
• Surgical Tattoos in Infrared: A Dataset for Quantifying Tissue Tracking and Mapping
• Creating and troubleshooting microscopy analysis workflows: common challenges and common solutions
• Tissue Concepts: supervised foundation models in computational pathology
• Enhancing Cell Instance Segmentation in Scanning Electron Microscopy Images via a Deep Contour Closing Operator
• Advances in Automated Fetal Brain MRI Segmentation and Biometry: Insights from the FeTA 2024 Challenge
• Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge
• Anatomical feature-prioritized loss for enhanced MR to CT translation
• Mask of truth: model sensitivity to unexpected regions of medical images
• Application-driven Validation of Posteriors in Inverse Problems
• Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets
• Automated neuroradiological support systems for multiple cerebrovascular disease markers -- A systematic review and meta-analysis
• Sex-based Bias Inherent in the Dice Similarity Coefficient: A Model Independent Analysis for Multiple Anatomical Structures
• KEVS: Enhancing Segmentation of Visceral Adipose Tissue in Pre-Cystectomy CT with Gaussian Kernel Density Estimation
• Improve Cross-Modality Segmentation by Treating T1-Weighted MRI Images as Inverted CT Scans
• Super-resolution multi-contrast unbiased eye atlases with deep probabilistic refinement
• AI Powered Road Network Prediction with Multi-Modal Data
• First Investigation of Deep Learning for Intraoperative Gauze Segmentation in Minimally Invasive Abdominal Surgery
• MonoUNet: A Robust Tiny Neural Network for Automated Knee Cartilage Segmentation on Point-of-Care Ultrasound Devices
• ROIsGAN: A Region Guided Generative Adversarial Framework for Murine Hippocampal Subregion Segmentation