Learning Domain Invariant Representations by Joint Wasserstein Distance Minimization

Abstract

Domain shifts in the training data are common in practical applications of machine learning; they occur for instance when the data is coming from different sources. Ideally, a ML model should work well independently of these shifts, for example, by learning a domain-invariant representation. However, common ML losses do not give strong guarantees on how consistently the ML model performs for different domains, in particular, whether the model performs well on a domain at the expense of its performance on another domain. In this paper, we build new theoretical foundations for this problem, by contributing a set of mathematical relations between classical losses for supervised ML and the Wasserstein distance in joint space (i.e. representation and output space). We show that classification or regression losses, when combined with a GAN-type discriminator between domains, form an upper-bound to the true Wasserstein distance between domains. This implies a more invariant representation and also more stable prediction performance across domains. Theoretical results are corroborated empirically on several image datasets. Our proposed approach systematically produces the highest minimum classification accuracy across domains, and the most invariant representation.

arXiv comment23 pages + supplement

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Spectral Normalization for Generative Adversarial Networks
• Improved Regularization of Convolutional Neural Networks with Cutout
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Domain Generalization: A Survey
• Semi-Supervised Learning with Ladder Networks
• Domain Generalization via Invariant Feature Representation
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Triple Generative Adversarial Nets
• Generalizing to unseen domains via distribution matching
• Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization
• Domain Generalization via Optimal Transport with Metric Similarity Learning
• Virtual Mixup Training for Unsupervised Domain Adaptation
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• Domain Generalization with MixStyle
• Semi-Supervised Domain Adaptation with Non-Parametric Copulas
• Domain Generalization via Semi-supervised Meta Learning
• Bures Joint Distribution Alignment with Dynamic Margin for Unsupervised Domain Adaptation

Cited by in corpus (1)

• Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI