Mind the Gap: Federated Learning Broadens Domain Generalization in Diagnostic AI Models

Abstract

Developing robust artificial intelligence (AI) models that generalize well to unseen datasets is challenging and usually requires large and variable datasets, preferably from multiple institutions. In federated learning (FL), a model is trained collaboratively at numerous sites that hold local datasets without exchanging them. So far, the impact of training strategy, i.e., local versus collaborative, on the diagnostic on-domain and off-domain performance of AI models interpreting chest radiographs has not been assessed. Consequently, using 610,000 chest radiographs from five institutions across the globe, we assessed diagnostic performance as a function of training strategy (i.e., local vs. collaborative), network architecture (i.e., convolutional vs. transformer-based), generalization performance (i.e., on-domain vs. off-domain), imaging finding (i.e., cardiomegaly, pleural effusion, pneumonia, atelectasis, consolidation, pneumothorax, and no abnormality), dataset size (i.e., from n=18,000 to 213,921 radiographs), and dataset diversity. Large datasets not only showed minimal performance gains with FL but, in some instances, even exhibited decreases. In contrast, smaller datasets revealed marked improvements. Thus, on-domain performance was mainly driven by training data size. However, off-domain performance leaned more on training diversity. When trained collaboratively across diverse external institutions, AI models consistently surpassed models trained locally for off-domain tasks, emphasizing FL's potential in leveraging data diversity. In conclusion, FL can bolster diagnostic privacy, reproducibility, and off-domain reliability of AI models and, potentially, optimize healthcare outcomes.

arXiv commentPublished in Nature Scientific Reports

References in corpus (12)

• A Survey on Deep Learning in Medical Image Analysis
• ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
• Federated Learning: Strategies for Improving Communication Efficiency
• Confounding variables can degrade generalization performance of radiological deep learning models
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Deep Learning Scaling is Predictable, Empirically
• Deep ROC Analysis and AUC as Balanced Average Accuracy to Improve Model Selection, Understanding and Interpretation
• Label-Efficient Self-Supervised Federated Learning for Tackling Data Heterogeneity in Medical Imaging
• Can we trust deep learning models diagnosis? The impact of domain shift in chest radiograph classification
• Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images
• Collaborative Training of Medical Artificial Intelligence Models with non-uniform Labels
• Federated learning for secure development of AI models for Parkinson's disease detection using speech from different languages

Cited by in corpus (2)

• Boosting multi-demographic federated learning for chest radiograph analysis using general-purpose self-supervised representations
• Resolution scaling governs DINOv3 transfer performance in chest radiograph classification