Model-Based Domain Generalization
arXiv:2102.11436
Abstract
Despite remarkable success in a variety of applications, it is well-known that deep learning can fail catastrophically when presented with out-of-distribution data. Toward addressing this challenge, we consider the domain generalization problem, wherein predictors are trained using data drawn from a family of related training domains and then evaluated on a distinct and unseen test domain. We show that under a natural model of data generation and a concomitant invariance condition, the domain generalization problem is equivalent to an infinite-dimensional constrained statistical learning problem; this problem forms the basis of our approach, which we call Model-Based Domain Generalization. Due to the inherent challenges in solving constrained optimization problems in deep learning, we exploit nonconvex duality theory to develop unconstrained relaxations of this statistical problem with tight bounds on the duality gap. Based on this theoretical motivation, we propose a novel domain generalization algorithm with convergence guarantees. In our experiments, we report improvements of up to 30 percentage points over state-of-the-art domain generalization baselines on several benchmarks including ColoredMNIST, Camelyon17-WILDS, FMoW-WILDS, and PACS.
References in corpus (61)
- Adam: A Method for Stochastic Optimization
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- mixup: Beyond Empirical Risk Minimization
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Towards Deep Learning Models Resistant to Adversarial Attacks
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning
- Frustratingly Easy Domain Adaptation
- Provable defenses against adversarial examples via the convex outer adversarial polytope
- Domain Generalization via Invariant Feature Representation
- AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
- Domain Generalization via Model-Agnostic Learning of Semantic Features
- Generalizing to Unseen Domains via Adversarial Data Augmentation
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- Generalizing Across Domains via Cross-Gradient Training
- Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
- Invariant Risk Minimization
- Domain Adaptation for Visual Applications: A Comprehensive Survey
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Towards a Definition of Disentangled Representations
- Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- Domain Generalization by Marginal Transfer Learning
- Improve Unsupervised Domain Adaptation with Mixup Training
- Heterogeneous Domain Generalization via Domain Mixup
- Does Distributionally Robust Supervised Learning Give Robust Classifiers?
- Learning Robust Representations by Projecting Superficial Statistics Out
- The Risks of Invariant Risk Minimization
- DIVA: Domain Invariant Variational Autoencoders
- Improving the Robustness of Deep Neural Networks via Stability Training
- Generalizing to unseen domains via distribution matching
- In Search of Lost Domain Generalization
- Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization
- Learning Invariant Representations for Reinforcement Learning without Reconstruction
- Feature-Critic Networks for Heterogeneous Domain Generalization
- DC3: A learning method for optimization with hard constraints
- Noise or Signal: The Role of Image Backgrounds in Object Recognition
- Understanding the Failure Modes of Out-of-Distribution Generalization
- Domain Generalization via Multidomain Discriminant Analysis
- Self-Challenging Improves Cross-Domain Generalization
- No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
- Learning perturbation sets for robust machine learning
- When Unseen Domain Generalization is Unnecessary? Rethinking Data Augmentation
- Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
- Learning Weighted Representations for Generalization Across Designs
- Nonlinear Invariant Risk Minimization: A Causal Approach
- Does Invariant Risk Minimization Capture Invariance?
- Constrained Learning with Non-Convex Losses
- A Generalization Error Bound for Multi-class Domain Generalization
- BREEDS: Benchmarks for Subpopulation Shift
- Towards Shape Biased Unsupervised Representation Learning for Domain Generalization
- Medical Image Harmonization Using Deep Learning Based Canonical Mapping: Toward Robust and Generalizable Learning in Imaging
- Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
- Gradient Matching for Domain Generalization
- Robust Reinforcement Learning using Adversarial Populations
- Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning
- Constrained Deep Learning using Conditional Gradient and Applications in Computer Vision
- A Strong Baseline for Domain Adaptation and Generalization in Medical Imaging
- Accounting for Unobserved Confounding in Domain Generalization
- FOCUS: Familiar Objects in Common and Uncommon Settings