Gradient Matching for Domain Generalization

Abstract

Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their ability to generalize to unseen domains. Here, we propose an inter-domain gradient matching objective that targets domain generalization by maximizing the inner product between gradients from different domains. Since direct optimization of the gradient inner product can be computationally prohibitive -- requires computation of second-order derivatives -- we derive a simpler first-order algorithm named Fish that approximates its optimization. We demonstrate the efficacy of Fish on 6 datasets from the Wilds benchmark, which captures distribution shift across a diverse range of modalities. Our method produces competitive results on these datasets and surpasses all baselines on 4 of them. We perform experiments on both the Wilds benchmark, which captures distribution shift in the real world, as well as datasets in DomainBed benchmark that focuses more on synthetic-to-real transfer. Our method produces competitive results on both benchmarks, demonstrating its effectiveness across a wide range of domain generalization tasks.

References in corpus (14)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Theoretical Models of Learning to Learn
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• On First-Order Meta-Learning Algorithms
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Invariant Risk Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Fairness Without Demographics in Repeated Loss Minimization
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Improve Unsupervised Domain Adaptation with Mixup Training
• Does Distributionally Robust Supervised Learning Give Robust Classifiers?
• In Search of Lost Domain Generalization
• Self-Challenging Improves Cross-Domain Generalization

Cited by in corpus (6)

• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• Model-Based Domain Generalization
• Accounting for Unobserved Confounding in Domain Generalization
• Learning Representations that Support Robust Transfer of Predictors
• Towards Data-Free Domain Generalization