On the limits of cross-domain generalization in automated X-ray prediction

Abstract

This large scale study focuses on quantifying what X-rays diagnostic prediction tasks generalize well across multiple different datasets. We present evidence that the issue of generalization is not due to a shift in the images but instead a shift in the labels. We study the cross-domain performance, agreement between models, and model representations. We find interesting discrepancies between performance and agreement where models which both achieve good performance disagree in their predictions as well as models which agree yet achieve poor performance. We also test for concept similarity by regularizing a network to group tasks across multiple datasets together and observe variation across the tasks. All code is made available online and data is publicly available: https://github.com/mlmed/torchxrayvision

arXiv commentFull paper at MIDL2020

References in corpus (10)

• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
• Confounding variables can degrade generalization performance of radiological deep learning models
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Large Scale Automated Reading of Frontal and Lateral Chest X-Rays using Dual Convolutional Neural Networks
• TorchXRayVision: A library of chest X-ray datasets and models
• Chester: A Web Delivered Locally Computed Chest X-Ray Disease Prediction System
• Quantifying the Value of Lateral Views in Deep Learning for Chest X-rays
• Do Lateral Views Help Automated Chest X-ray Predictions?

Cited by in corpus (11)

• COVID-19 Image Data Collection: Prospective Predictions Are the Future
• Deep Learning for Chest X-ray Analysis: A Survey
• Domain Generalization using Causal Matching
• TorchXRayVision: A library of chest X-ray datasets and models
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays
• Pay Attention to the cough: Early Diagnosis of COVID-19 using Interpretable Symptoms Embeddings with Cough Sound Signal Processing
• On the Composition and Limitations of Publicly Available COVID-19 X-Ray Imaging Datasets
• Learning from Multiple Datasets with Heterogeneous and Partial Labels for Universal Lesion Detection in CT
• Universal Lesion Detection by Learning from Multiple Heterogeneously Labeled Datasets
• An Empirical Framework for Domain Generalization in Clinical Settings