How Many Annotators Do We Need? -- A Study on the Influence of Inter-Observer Variability on the Reliability of Automatic Mitotic Figure Assessment

Abstract

Density of mitotic figures in histologic sections is a prognostically relevant characteristic for many tumours. Due to high inter-pathologist variability, deep learning-based algorithms are a promising solution to improve tumour prognostication. Pathologists are the gold standard for database development, however, labelling errors may hamper development of accurate algorithms. In the present work we evaluated the benefit of multi-expert consensus (n = 3, 5, 7, 9, 11) on algorithmic performance. While training with individual databases resulted in highly variable F${}_1$ scores, performance was notably increased and more consistent when using the consensus of three annotators. Adding more annotators only resulted in minor improvements. We conclude that databases by few pathologists and high label accuracy may be the best compromise between high algorithmic performance and time investment.

arXiv commentDue to data inconsistencies experiments had to be repeated with a reduced number of annotators (17 in version 1). All findings of the previous version were reproducible. 7 pages, 2 figures, accepted at BVM workshop 2021

References in corpus (5)

• Assessment of algorithms for mitosis detection in breast cancer histopathology images
• Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge
• Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks
• Deep learning algorithms out-perform veterinary pathologists in detecting the mitotically most active tumor region
• Are pathologist-defined labels reproducible? Comparison of the TUPAC16 mitotic figure dataset with an alternative set of labels

Cited by in corpus (2)

• Pan-tumor CAnine cuTaneous Cancer Histology (CATCH) dataset
• On the Value of PHH3 for Mitotic Figure Detection on H&E-stained Images