Understanding metric-related pitfalls in image analysis validation

Abstract

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Shared first authors: Annika Reinke and Minu D. Tizabi; shared senior authors: Lena Maier-Hein and Paul F. Jäger. Published in Nature Methods. arXiv admin note: text overlap with arXiv:2206.01653

References in corpus (8)

• Cell Detection with Star-convex Polygons
• The Liver Tumor Segmentation Benchmark (LiTS)
• Metrics reloaded: Recommendations for image analysis validation
• Why rankings of biomedical image analysis competitions should be interpreted with care
• ROC and AUC with a Binary Predictor: a Potentially Misleading Metric
• Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient
• Better Uncertainty Calibration via Proper Scores for Classification and Beyond
• Sources of performance variability in deep learning-based polyp detection

Cited by in corpus (9)

• Metrics reloaded: Recommendations for image analysis validation
• Similarity and Quality Metrics for MR Image-To-Image Translation
• Similarity of Neural Network Models: A Survey of Functional and Representational Measures
• A Robust Ensemble Algorithm for Ischemic Stroke Lesion Segmentation: Generalizability and Clinical Utility Beyond the ISLES Challenge
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge
• The ATLAS Virtual Research Assistant
• Application-driven Validation of Posteriors in Inverse Problems
• Fréchet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets