Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging

Abstract

We reproduced the results of CheXNet with fixed hyperparameters and 50 different random seeds to identify 14 finding in chest radiographs (x-rays). Because CheXNet fine-tunes a pre-trained DenseNet, the random seed affects the ordering of the batches of training data but not the initialized model weights. We found substantial variability in predictions for the same radiograph across model runs (mean ln[(maximum probability)/(minimum probability)] 2.45, coefficient of variation 0.543). This individual radiograph-level variability was not fully reflected in the variability of AUC on a large test set. Averaging predictions from 10 models reduced variability by nearly 70% (mean coefficient of variation from 0.543 to 0.169, t-test 15.96, p-value < 0.0001). We encourage researchers to be aware of the potential variability of CNNs and ensemble predictions from multiple models to minimize the effect this variability may have on the care of individual patients when these models are deployed clinically.

arXiv commentJ.Z. and J.F. contributed equally to this work

References in corpus (10)

• Confounding variables can degrade generalization performance of radiological deep learning models
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Reinforcement Learning that Matters
• On Empirical Comparisons of Optimizers for Deep Learning
• Bayesian Recurrent Neural Networks
• Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening
• Detecting hip fractures with radiologist-level performance using deep neural networks
• Direct Uncertainty Prediction for Medical Second Opinions
• The Scientific Method in the Science of Machine Learning