Quantifying and Improving Transferability in Domain Generalization

Abstract

Out-of-distribution generalization is one of the key challenges when transferring a model from the lab to the real world. Existing efforts mostly focus on building invariant features among source and target domains. Based on invariant features, a high-performing classifier on source domains could hopefully behave equally well on a target domain. In other words, the invariant features are \emph{transferable}. However, in practice, there are no perfectly transferable features, and some algorithms seem to learn "more transferable" features than others. How can we understand and quantify such \emph{transferability}? In this paper, we formally define transferability that one can quantify and compute in domain generalization. We point out the difference and connection with common discrepancy measures between domains, such as total variation and Wasserstein distance. We then prove that our transferability can be estimated with enough samples and give a new upper bound for the target error based on our transferability. Empirically, we evaluate the transferability of the feature embeddings learned by existing algorithms for domain generalization. Surprisingly, we find that many algorithms are not quite learning transferable features, although few could still survive. In light of this, we propose a new algorithm for learning transferable features and test it over various benchmark datasets, including RotatedMNIST, PACS, Office-Home and WILDS-FMoW. Experimental results show that the proposed algorithm achieves consistent improvement over many state-of-the-art algorithms, corroborating our theoretical findings.

arXiv commentNeurIPS 2021

References in corpus (14)

• Certified Adversarial Robustness via Randomized Smoothing
• Domain Generalization via Invariant Feature Representation
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Invariant Risk Minimization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Generalization and Equilibrium in Generative Adversarial Nets (GANs)
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Improve Unsupervised Domain Adaptation with Mixup Training
• Generalizing to unseen domains via distribution matching
• In Search of Lost Domain Generalization
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• When is invariance useful in an Out-of-Distribution Generalization problem ?