Confounding variables can degrade generalization performance of radiological deep learning models

Abstract

Early results in using convolutional neural networks (CNNs) on x-rays to diagnose disease have been promising, but it has not yet been shown that models trained on x-rays from one hospital or one group of hospitals will work equally well at different hospitals. Before these tools are used for computer-aided diagnosis in real-world clinical settings, we must verify their ability to generalize across a variety of hospital systems. A cross-sectional design was used to train and evaluate pneumonia screening CNNs on 158,323 chest x-rays from NIH (n=112,120 from 30,805 patients), Mount Sinai (42,396 from 12,904 patients), and Indiana (n=3,807 from 3,683 patients). In 3 / 5 natural comparisons, performance on chest x-rays from outside hospitals was significantly lower than on held-out x-rays from the original hospital systems. CNNs were able to detect where an x-ray was acquired (hospital system, hospital department) with extremely high accuracy and calibrate predictions accordingly. The performance of CNNs in diagnosing diseases on x-rays may reflect not only their ability to identify disease-specific imaging findings on x-rays, but also their ability to exploit confounding information. Estimates of CNN performance based on test data from hospital systems used for model training may overstate their likely real-world performance.

References in corpus (6)

• Scikit-learn: Machine Learning in Python
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Understanding deep learning requires rethinking generalization
• High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Detecting hip fractures with radiologist-level performance using deep neural networks

Cited by in corpus (144)

• Shortcut Learning in Deep Neural Networks
• An overview of deep learning in medical imaging focusing on MRI
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem
• Ethical Machine Learning in Health Care
• Deep Learning for Chest X-ray Analysis: A Survey
• Federated Learning Enables Big Data for Rare Cancer Boundary Detection
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Diagnosis of Coronavirus Disease 2019 (COVID-19) with Structured Latent Multi-View Representation Learning
• Contrastive Cross-site Learning with Redesigned Net for COVID-19 CT Classification
• Radiological images and machine learning: trends, perspectives, and prospects
• Uncertainty and Interpretability in Convolutional Neural Networks for Semantic Segmentation of Colorectal Polyps
• The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study
• MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Deep Learning for Automatic Pneumonia Detection
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• OpenFL: An open-source framework for Federated Learning
• Expert-Augmented Machine Learning
• From Bit To Bedside: A Practical Framework For Artificial Intelligence Product Development In Healthcare
• Harmony-Search and Otsu based System for Coronavirus Disease (COVID-19) Detection using Lung CT Scan Images
• Detecting Shortcut Learning for Fair Medical AI using Shortcut Testing
• What Do Compressed Deep Neural Networks Forget?
• The challenges of deploying artificial intelligence models in a rapidly evolving pandemic
• Data Valuation for Medical Imaging Using Shapley Value: Application on A Large-scale Chest X-ray Dataset
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• On the limits of cross-domain generalization in automated X-ray prediction
• Ensembles of Convolutional Neural Networks models for pediatric pneumonia diagnosis
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• Self-Supervised Pre-Training with Contrastive and Masked Autoencoder Methods for Dealing with Small Datasets in Deep Learning for Medical Imaging
• Domain generalization in deep learning-based mass detection in mammography: A large-scale multi-center study
• Ten Quick Tips for Deep Learning in Biology
• On Interpretability of Artificial Neural Networks: A Survey
• Augmenting deep neural networks with symbolic knowledge: Towards trustworthy and interpretable AI for education
• Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Tutorial: Safe and Reliable Machine Learning
• Cross-Site Severity Assessment of COVID-19 from CT Images via Domain Adaptation
• Improving Clinician Performance in Classification of EEG Patterns on the Ictal-Interictal-Injury Continuum using Interpretable Machine Learning
• Domain-guided data augmentation for deep learning on medical imaging
• The Scientific Method in the Science of Machine Learning
• Vision Transformer for COVID-19 CXR Diagnosis using Chest X-ray Feature Corpus
• NCI Workshop on Artificial Intelligence in Radiation Oncology: Training the Next Generation
• Risk of AI in Healthcare: A Comprehensive Literature Review and Study Framework
• Operational Calibration: Debugging Confidence Errors for DNNs in the Field
• Uncertainty-based Modulation for Lifelong Learning
• Mind the Gap: Federated Learning Broadens Domain Generalization in Diagnostic AI Models
• Explainable multiple abnormality classification of chest CT volumes
• CheXpedition: Investigating Generalization Challenges for Translation of Chest X-Ray Algorithms to the Clinical Setting
• A DICOM Framework for Machine Learning Pipelines against Real-Time Radiology Images
• Multi-Center Fetal Brain Tissue Annotation (FeTA) Challenge 2022 Results
• CosSIF: Cosine similarity-based image filtering to overcome low inter-class variation in synthetic medical image datasets
• "The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep Learning
• Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays
• Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
• A Unifying Causal Framework for Analyzing Dataset Shift-stable Learning Algorithms
• Deep learning classification of chest x-ray images
• Early detection of sepsis utilizing deep learning on electronic health record event sequences
• Counterfactuals and Causability in Explainable Artificial Intelligence: Theory, Algorithms, and Applications
• Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systems
• IGUANe: a 3D generalizable CycleGAN for multicenter harmonization of brain MR images
• In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review
• The Federated Tumor Segmentation (FeTS) Challenge
• Exploring the Efficacy of Base Data Augmentation Methods in Deep Learning-Based Radiograph Classification of Knee Joint Osteoarthritis
• Deep learning-based prediction of response to HER2-targeted neoadjuvant chemotherapy from pre-treatment dynamic breast MRI: A multi-institutional validation study
• IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography
• FetMRQC: a robust quality control system for multi-centric fetal brain MRI
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• CheXternal: Generalization of Deep Learning Models for Chest X-ray Interpretation to Photos of Chest X-rays and External Clinical Settings
• The Cells Out of Sample (COOS) dataset and benchmarks for measuring out-of-sample generalization of image classifiers
• Fighting the scanner effect in brain MRI segmentation with a progressive level-of-detail network trained on multi-site data
• Measuring the effects of confounders in medical supervised classification problems: the Confounding Index (CI)
• Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks
• Removing the influence of a group variable in high-dimensional predictive modelling
• Spectral decoupling allows training transferable neural networks in medical imaging
• Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport
• On the Composition and Limitations of Publicly Available COVID-19 X-Ray Imaging Datasets
• A Human-Centered Machine-Learning Approach for Muscle-Tendon Junction Tracking in Ultrasound Images
• Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Features
• Deep learning in magnetic resonance prostate segmentation: A review and a new perspective
• Further Analysis of Outlier Detection with Deep Generative Models
• Optimal Sparse Decision Trees
• DeepAAA: clinically applicable and generalizable detection of abdominal aortic aneurysm using deep learning
• Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification
• Now You See It, Now You Dont: Adversarial Vulnerabilities in Computational Pathology
• Multi-Task Learning in Histo-pathology for Widely Generalizable Model
• Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
• I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models
• Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician
• Improve Model Generalization and Robustness to Dataset Bias with Bias-regularized Learning and Domain-guided Augmentation
• A Strong Baseline for Domain Adaptation and Generalization in Medical Imaging
• Deep Mining External Imperfect Data for Chest X-ray Disease Screening
• Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts
• Accounting for Unobserved Confounding in Domain Generalization
• Validation of a deep learning mammography model in a population with low screening rates
• Multi-source-free Domain Adaptation via Uncertainty-aware Adaptive Distillation
• Learning Deep Attribution Priors Based On Prior Knowledge
• Interpretable and synergistic deep learning for visual explanation and statistical estimations of segmentation of disease features from medical images
• Was there COVID-19 back in 2012? Challenge for AI in Diagnosis with Similar Indications
• M3Lung-Sys: A Deep Learning System for Multi-Class Lung Pneumonia Screening from CT Imaging
• Dimensionality Reduction for Improving Out-of-Distribution Detection in Medical Image Segmentation
• Global Saliency: Aggregating Saliency Maps to Assess Dataset Artefact Bias
• ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries
• Challenges for cognitive decoding using deep learning methods
• What If We Simply Swap the Two Text Fragments? A Straightforward yet Effective Way to Test the Robustness of Methods to Confounding Signals in Nature Language Inference Tasks
• Adversarial Scrutiny of Evidentiary Statistical Software
• Disrupting Model Training with Adversarial Shortcuts
• Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• An Information-theoretic Approach to Distribution Shifts
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors
• The unreasonable effectiveness of Batch-Norm statistics in addressing catastrophic forgetting across medical institutions
• VIS-MAE: An Efficient Self-supervised Learning Approach on Medical Image Segmentation and Classification
• Assessing Generalization Capabilities of Malaria Diagnostic Models from Thin Blood Smears
• Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
• Causal bootstrapping
• Disease classification of macular Optical Coherence Tomography scans using deep learning software: validation on independent, multi-centre data
• The Utility of General Domain Transfer Learning for Medical Language Tasks
• Decomposition of Global Feature Importance into Direct and Associative Components (DEDACT)
• Do Input Gradients Highlight Discriminative Features?
• Explainable Machine Learning in Healthcare: Methods, Interpretation, and Applications for Clinical Research
• Towards Robust Classification Model by Counterfactual and Invariant Data Generation
• Failing Conceptually: Concept-Based Explanations of Dataset Shift
• Structured dataset documentation: a datasheet for CheXpert
• Addressing Fairness, Bias and Class Imbalance in Machine Learning: the FBI-loss
• Survey of the Detection and Classification of Pulmonary Lesions via CT and X-Ray
• Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
• Artificial Intelligence for Pediatric Ophthalmology
• A Multisite, Report-Based, Centralized Infrastructure for Feedback and Monitoring of Radiology AI/ML Development and Clinical Deployment
• A Standardized Radiograph-Agnostic Framework and Platform For Evaluating AI Radiological Systems
• Minimax Optimal Estimation of Stability Under Distribution Shift
• The Pragmatic Frames of Spurious Correlations in Machine Learning: Interpreting How and Why They Matter
• Comparative Analysis of Deep Learning Architectures for Multi-Disease Classification of Single-Label Chest X-rays
• Advancement of Deep Learning in Pneumonia and Covid-19 Classification and Localization: A Qualitative and Quantitative Analysis
• An Empirical Framework for Domain Generalization in Clinical Settings
• Unmasking Biases and Reliability Concerns in Convolutional Neural Networks Analysis of Cancer Pathology Images
• FPAENet: Pneumonia Detection Network Based on Feature Pyramid Attention Enhancement
• Requirement analysis for an artificial intelligence model for the diagnosis of the COVID-19 from chest X-ray data
• Artificial Intelligence Enhanced Rapid and Efficient Diagnosis of Mycoplasma Pneumoniae Pneumonia in Children Patients
• Generalization of Deep Convolutional Neural Networks -- A Case-study on Open-source Chest Radiographs