Common Limitations of Image Processing Metrics: A Picture Story

Abstract

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

arXiv commentShared first authors: Annika Reinke and Minu D. Tizabi. This is a dynamic paper on limitations of commonly used metrics. It discusses metrics for image-level classification, semantic and instance segmentation, and object detection. For missing use cases, comments or questions, please contact [email protected]. Substantial contributions to this document will be acknowledged with a co-authorship

References in corpus (1)

• Why rankings of biomedical image analysis competitions should be interpreted with care

Cited by in corpus (13)

• The Medical Segmentation Decathlon
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
• Robust deep learning-based semantic organ segmentation in hyperspectral images
• Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• A deep learning framework for the detection and quantification of drusen and reticular pseudodrusen on optical coherence tomography
• Semantic segmentation of multispectral photoacoustic images using deep learning
• The TopCoW Challenge -- Topology-Aware Circle of Willis Segmentation for CT and MR Angiography
• Revisiting Hidden Representations in Transfer Learning for Medical Imaging
• Architecture Analysis and Benchmarking of 3D U-shaped Deep Learning Models for Thoracic Anatomical Segmentation
• CT respiratory motion synthesis using joint supervised and adversarial learning
• Chameleon: A Semi-AutoML framework targeting quick and scalable development and deployment of production-ready ML systems for SMEs
• Why is the winner the best?