Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans
arXiv:2008.06388 · doi:10.1038/s42256-021-00307-0
Abstract
Machine learning methods offer great promise for fast and accurate detection and prognostication of COVID-19 from standard-of-care chest radiographs (CXR) and computed tomography (CT) images. Many articles have been published in 2020 describing new machine learning-based models for both of these tasks, but it is unclear which are of potential clinical utility. In this systematic review, we search EMBASE via OVID, MEDLINE via PubMed, bioRxiv, medRxiv and arXiv for published papers and preprints uploaded from January 1, 2020 to October 3, 2020 which describe new machine learning models for the diagnosis or prognosis of COVID-19 from CXR or CT images. Our search identified 2,212 studies, of which 415 were included after initial screening and, after quality screening, 61 studies were included in this systematic review. Our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases. This is a major weakness, given the urgency with which validated COVID-19 models are needed. To address this, we give many recommendations which, if followed, will solve these issues and lead to higher quality model development and well documented manuscripts.
35 pages, 3 figures, 2 tables, updated to the period 1 January 2020 - 3 October 2020
References in corpus (4)
- Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation and Diagnosis for COVID-19
- Improving performance of CNN to predict likelihood of COVID-19 using chest X-ray images with preprocessing algorithms
- COVID-19 in CXR: from Detection and Severity Scoring to Patient Disease Monitoring
- Was there COVID-19 back in 2012? Challenge for AI in Diagnosis with Similar Indications
Cited by in corpus (36)
- Recent advances and clinical applications of deep learning in medical image analysis
- How I failed machine learning in medical imaging -- shortcomings and recommendations
- Artificial intelligence in digital pathology: a systematic review and meta-analysis of diagnostic test accuracy
- The Fallacy of AI Functionality
- Technology Readiness Levels for Machine Learning Systems
- A brief history of AI: how to prevent another winter (a critical review)
- Artificial intelligence adoption in the physical sciences, natural sciences, life sciences, social sciences and the arts and humanities: A bibliometric analysis of research publications from 1960-2021
- Classification of datasets with imputed missing values: does imputation quality matter?
- How to avoid machine learning pitfalls: a guide for academic researchers
- Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations
- Empowering Local Communities Using Artificial Intelligence
- Machine Learning on the COVID-19 Pandemic, Human Mobility and Air Quality: A Review
- Don't Push the Button! Exploring Data Leakage Risks in Machine Learning and Transfer Learning
- AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learning
- Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
- Multimodal Explainability via Latent Shift applied to COVID-19 stratification
- Highdicom: A Python library for standardized encoding of image annotations and machine learning model outputs in pathology and radiology
- MedalCare-XL: 16,900 healthy and pathological 12 lead ECGs obtained through electrophysiological simulations
- Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
- DC-Check: A Data-Centric AI checklist to guide the development of reliable machine learning systems
- Quickest Detection of COVID-19 Pandemic Onset
- FetMRQC: a robust quality control system for multi-centric fetal brain MRI
- RadEdit: stress-testing biomedical vision models via diffusion image editing
- Accelerating Electronic Stopping Power Predictions by 10 Million Times with a Combination of Time-Dependent Density Functional Theory and Machine Learning
- "It depends": Configuring AI to Improve Clinical Usefulness Across Contexts
- Navigating the challenges in creating complex data systems: a development philosophy
- Data Heterogeneity Modeling for Trustworthy Machine Learning
- Hybrid Quantum Machine Learning Assisted Classification of COVID-19 from Computed Tomography Scans
- Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
- On the Impact of Voice Anonymization on Speech Diagnostic Applications: a Case Study on COVID-19 Detection
- A Prospective Observational Study to Investigate Performance of a Chest X-ray Artificial Intelligence Diagnostic Support Tool Across 12 U.S. Hospitals
- Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
- On Control of Epidemics with Application to COVID-19
- CIRCA: comprehensible online system in support of chest X-rays-based COVID-19 diagnosis
- Requirement analysis for an artificial intelligence model for the diagnosis of the COVID-19 from chest X-ray data
- Data Requirement Goal Modeling for Machine Learning Systems