CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting

Abstract

Although there have been several recent advances in the application of deep learning algorithms to chest x-ray interpretation, we identify three major challenges for the translation of chest x-ray algorithms to the clinical setting. We examine the performance of the top 10 performing models on the CheXpert challenge leaderboard on three tasks: (1) TB detection, (2) pathology detection on photos of chest x-rays, and (3) pathology detection on data from an external institution. First, we find that the top 10 chest x-ray models on the CheXpert competition achieve an average AUC of 0.851 on the task of detecting TB on two public TB datasets without fine-tuning or including the TB labels in training data. Second, we find that the average performance of the models on photos of x-rays (AUC = 0.916) is similar to their performance on the original chest x-ray images (AUC = 0.924). Third, we find that the models tested on an external dataset either perform comparably to or exceed the average performance of radiologists. We believe that our investigation will inform rapid translation of deep learning algorithms to safe and effective clinical decision support tools that can be validated prospectively with large impact studies and clinical trials.

arXiv commentAccepted as workshop paper at ACM Conference on Health, Inference, and Learning (CHIL) 2020

References in corpus (5)

• Confounding variables can degrade generalization performance of radiological deep learning models
• Adversarially Robust Generalization Requires More Data
• MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs
• Learning Deep Features for Discriminative Localization
• Deep Learning for the Digital Pathologic Diagnosis of Cholangiocarcinoma and Hepatocellular Carcinoma: Evaluating the Impact of a Web-based Diagnostic Assistant

Cited by in corpus (10)

• Supervised Transfer Learning at Scale for Medical Imaging
• CheXphoto: 10,000+ Photos and Transformations of Chest X-rays for Benchmarking Deep Learning Robustness
• CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings
• Generalizability issues with deep learning models in medicine and their potential solutions: illustrated with Cone-Beam Computed Tomography (CBCT) to Computed Tomography (CT) image conversion
• No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets
• CheXphotogenic: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays
• A clinical validation of VinDr-CXR, an AI system for detecting abnormal chest radiographs
• Learning to diagnose common thorax diseases on chest radiographs from radiology reports in Vietnamese
• CheXbreak: Misclassification Identification for Deep Learning Models Interpreting Chest X-rays
• CheXseen: Unseen Disease Detection for Deep Learning Interpretation of Chest X-rays