In Search of Lost Domain Generalization

Abstract

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

References in corpus (30)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• mixup: Beyond Empirical Risk Minimization
• Domain Generalization via Invariant Feature Representation
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Generalizing Across Domains via Cross-Gradient Training
• Invariant Risk Minimization
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• On Learning Invariant Representation for Domain Adaptation
• Invariant Models for Causal Transfer Learning
• Domain Generalization by Marginal Transfer Learning
• Improve Unsupervised Domain Adaptation with Mixup Training
• Heterogeneous Domain Generalization via Domain Mixup
• DIVA: Domain Invariant Variational Autoencoders
• Support and Invertibility in Domain-Invariant Representations
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• Unshuffling Data for Improved Generalization
• Domain Generalization via Conditional Invariant Representation
• Episodic Training for Domain Generalization
• Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Adversarial Invariant Feature Learning with Accuracy Constraint for Domain Generalization
• A Generalization Error Bound for Multi-class Domain Generalization
• Towards Shape Biased Unsupervised Representation Learning for Domain Generalization
• Adversarial Domain Adaptation with Domain Mixup
• Deep Domain-Adversarial Image Generation for Domain Generalisation
• Hidden Covariate Shift: A Minimal Assumption For Domain Adaptation
• Sequential Learning for Domain Generalization

Cited by in corpus (54)

• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Domain Generalization using Causal Matching
• Understanding the Failure Modes of Out-of-Distribution Generalization
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• Unshuffling Data for Improved Generalization
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Test-time Batch Statistics Calibration for Covariate Shift
• AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• Robust and Generalizable Visual Representation Learning via Random Convolutions
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
• Gradient Matching for Domain Generalization
• Multiple Domain Experts Collaborative Learning: Multi-Source Domain Generalization For Person Re-Identification
• Out-of-Distribution Generalization Analysis via Influence Function
• Simple data balancing achieves competitive worst-group-accuracy
• On Calibration and Out-of-domain Generalization
• Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• A Theory of Label Propagation for Subpopulation Shift
• Quantifying and Improving Transferability in Domain Generalization
• Model-Based Domain Generalization
• Domain Generalization for Session-Independent Brain-Computer Interface
• Accounting for Unobserved Confounding in Domain Generalization
• A Bit More Bayesian: Domain-Invariant Learning with Uncertainty
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• Learning Representations that Support Robust Transfer of Predictors
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• Robustness through Data Augmentation Loss Consistency
• An Information-theoretic Approach to Distribution Shifts
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• Learning Stable Classifiers by Transferring Unstable Features
• Robust Representation Learning via Perceptual Similarity Metrics
• Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers
• VideoDG: Generalizing Temporal Relations in Videos to Novel Domains
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
• Generalizing Nucleus Recognition Model in Multi-source Images via Pruning
• On Invariance Penalties for Risk Minimization
• Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
• KL Guided Domain Adaptation
• Task Guided Compositional Representation Learning for ZDA
• What classifiers know what they don't?
• Domain Conditional Predictors for Domain Adaptation
• Pseudo-domains in imaging data improve prediction of future disease status in multi-center studies
• Towards Data-Free Domain Generalization
• Reappraising Domain Generalization in Neural Networks
• An Empirical Framework for Domain Generalization in Clinical Settings
• What Can Style Transfer and Paintings Do For Model Robustness?