Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
arXiv:2010.04647
Abstract
The success of supervised learning hinges on the assumption that the training and test data come from the same underlying distribution, which is often not valid in practice due to potential distribution shift. In light of this, most existing methods for unsupervised domain adaptation focus on achieving domain-invariant representations and small source domain error. However, recent works have shown that this is not sufficient to guarantee good generalization on the target domain, and in fact, is provably detrimental under label distribution shift. Furthermore, in many real-world applications it is often feasible to obtain a small amount of labeled data from the target domain and use them to facilitate model training with source data. Inspired by the above observations, in this paper we propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA). First, we provide a finite sample bound for both classification and regression problems under Semi-DA. The bound suggests a principled way to obtain target generalization, i.e. by aligning both the marginal and conditional distributions across domains in feature space. Motivated by this, we then introduce the LIRR algorithm for jointly \textbf{L}earning \textbf{I}nvariant \textbf{R}epresentations and \textbf{R}isks. Finally, extensive experiments are conducted on both classification and regression tasks, which demonstrates LIRR consistently achieves state-of-the-art performance and significant improvements compared with the methods that only learn invariant representations or invariant risks.
Accepted to CVPR 2021
References in corpus (18)
- Learning Transferable Features with Deep Adaptation Networks
- Conditional Adversarial Domain Adaptation
- Domain Separation Networks
- VisDA: The Visual Domain Adaptation Challenge
- Invariant Risk Minimization
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- On Learning Invariant Representation for Domain Adaptation
- Multi-source Domain Adaptation for Semantic Segmentation
- Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
- Learning Controllable Fair Representations
- Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment
- A Minimax Approach to Supervised Learning
- Adversarial Dropout Regularization
- Cross-domain Self-supervised Learning for Domain Adaptation with Few Source Labels
- Attract, Perturb, and Explore: Learning a Feature Alignment Network for Semi-supervised Domain Adaptation
- Instance Adaptive Self-Training for Unsupervised Domain Adaptation
- On the Value of Target Data in Transfer Learning
- Fast Fair Regression via Efficient Approximations of Mutual Information