An Empirical Framework for Domain Generalization in Clinical Settings

Abstract

Clinical machine learning models experience significantly degraded performance in datasets not seen during training, e.g., new hospitals or populations. Recent developments in domain generalization offer a promising solution to this problem by creating models that learn invariances across environments. In this work, we benchmark the performance of eight domain generalization methods on multi-site clinical time series and medical imaging data. We introduce a framework to induce synthetic but realistic domain shifts and sampling bias to stress-test these methods over existing non-healthcare benchmarks. We find that current domain generalization methods do not consistently achieve significant gains in out-of-distribution performance over empirical risk minimization on real-world medical imaging data, in line with prior work on general imaging datasets. However, a subset of realistic induced-shift scenarios in clinical time series data do exhibit limited performance gains. We characterize these scenarios in detail, and recommend best practices for domain generalization in the clinical setting.

Published at ACM CHIL 2021

References in corpus (32)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Confounding variables can degrade generalization performance of radiological deep learning models
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Invariant Risk Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis
• The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study
• An Empirical Characterization of Fair Machine Learning For Clinical Risk Prediction
• Benchmarking machine learning models on multi-centre eICU critical care dataset
• The Risks of Invariant Risk Minimization
• In Search of Lost Domain Generalization
• On the limits of cross-domain generalization in automated X-ray prediction
• Anchor regression: heterogeneous data meets causality
• Efficient Domain Generalization via Common-Specific Low-Rank Decomposition
• A Closer Look at Domain Shift for Deep Learning in Histopathology
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• Invariant Risk Minimization Games
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Learning Invariances in Neural Networks
• Does Invariant Risk Minimization Capture Invariance?
• An Empirical Study of Invariant Risk Minimization
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Representation via Representations: Domain Generalization via Adversarially Learned Invariant Representations
• Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification
• Deeper, Broader and Artier Domain Generalization
• Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately
• Modeling the Biological Pathology Continuum with HSIC-regularized Wasserstein Auto-encoders
• Learning Diverse Representations for Fast Adaptation to Distribution Shift