In Search of Lost Domain Generalization

Abstract

The goal of domain generalization algorithms is to predict well on distributions different from those seen during training. While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions -- datasets, architectures, and model selection criteria -- render fair and realistic comparisons difficult. In this paper, we are interested in understanding how useful domain generalization algorithms are in realistic settings. As a first step, we realize that model selection is non-trivial for domain generalization tasks. Contrary to prior work, we argue that domain generalization algorithms without a model selection strategy should be regarded as incomplete. Next, we implement DomainBed, a testbed for domain generalization including seven multi-domain datasets, nine baseline algorithms, and three model selection criteria. We conduct extensive experiments using DomainBed and find that, when carefully implemented, empirical risk minimization shows state-of-the-art performance across all datasets. Looking forward, we hope that the release of DomainBed, along with contributions from fellow researchers, will streamline reproducible and rigorous research in domain generalization.

References in corpus (12)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Domain Generalization via Invariant Feature Representation
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• On Learning Invariant Representation for Domain Adaptation
• Improve Unsupervised Domain Adaptation with Mixup Training
• Heterogeneous Domain Generalization via Domain Mixup
• Support and Invertibility in Domain-Invariant Representations
• When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
• A Generalization Error Bound for Multi-class Domain Generalization
• Adversarial Domain Adaptation with Domain Mixup
• Sequential Learning for Domain Generalization

Cited by in corpus (24)

• Towards a Theoretical Framework of Out-of-Distribution Generalization
• Test-time Batch Statistics Calibration for Covariate Shift
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Out-of-Distribution Generalization Analysis via Influence Function
• Latent Adversarial Debiasing: Mitigating Collider Bias in Deep Neural Networks
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark
• Domain Generalization for Session-Independent Brain-Computer Interface
• Quantifying and Improving Transferability in Domain Generalization
• Learning Representations that Support Robust Transfer of Predictors
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• An Information-theoretic Approach to Distribution Shifts
• The Connection between Out-of-Distribution Generalization and Privacy of ML Models
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Robust Representation Learning via Perceptual Similarity Metrics
• Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers
• On Invariance Penalties for Risk Minimization
• Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
• Generalizing Nucleus Recognition Model in Multi-source Images via Pruning
• What classifiers know what they don't?
• Task Guided Compositional Representation Learning for ZDA
• An Empirical Framework for Domain Generalization in Clinical Settings
• Domain Conditional Predictors for Domain Adaptation