An Empirical Framework for Domain Generalization in Clinical Settings

Abstract

Clinical machine learning models experience significantly degraded performance in datasets not seen during training, e.g., new hospitals or populations. Recent developments in domain generalization offer a promising solution to this problem by creating models that learn invariances across environments. In this work, we benchmark the performance of eight domain generalization methods on multi-site clinical time series and medical imaging data. We introduce a framework to induce synthetic but realistic domain shifts and sampling bias to stress-test these methods over existing non-healthcare benchmarks. We find that current domain generalization methods do not consistently achieve significant gains in out-of-distribution performance over empirical risk minimization on real-world medical imaging data, in line with prior work on general imaging datasets. However, a subset of realistic induced-shift scenarios in clinical time series data do exhibit limited performance gains. We characterize these scenarios in detail, and recommend best practices for domain generalization in the clinical setting.

Published at ACM CHIL 2021

References in corpus (15)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis
• In Search of Lost Domain Generalization
• Efficient Domain Generalization via Common-Specific Low-Rank Decomposition
• A Closer Look at Domain Shift for Deep Learning in Histopathology
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• Learning Invariances in Neural Networks
• Does Invariant Risk Minimization Capture Invariance?
• Representation via Representations: Domain Generalization via Adversarially Learned Invariant Representations
• Fairness and Robustness in Invariant Learning: A Case Study in Toxicity Classification
• Deeper, Broader and Artier Domain Generalization
• Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionately
• Modeling the Biological Pathology Continuum with HSIC-regularized Wasserstein Auto-encoders
• Learning Diverse Representations for Fast Adaptation to Distribution Shift