On Learning Invariant Representation for Domain Adaptation

Abstract

Due to the ability of deep neural nets to learn rich representations, recent advances in unsupervised domain adaptation have focused on learning domain-invariant features that achieve a small error on the source domain. The hope is that the learnt representation, together with the hypothesis learnt from the source domain, can generalize to the target domain. In this paper, we first construct a simple counterexample showing that, contrary to common belief, the above conditions are not sufficient to guarantee successful domain adaptation. In particular, the counterexample exhibits \emph{conditional shift}: the class-conditional distributions of input features change between source and target domains. To give a sufficient condition for domain adaptation, we propose a natural and interpretable generalization upper bound that explicitly takes into account the aforementioned shift. Moreover, we shed new light on the problem by proving an information-theoretic lower bound on the joint error of \emph{any} domain adaptation method that attempts to learn invariant representations. Our result characterizes a fundamental tradeoff between learning invariant representations and achieving small joint error on both domains when the marginal label distributions differ from source to target. Finally, we conduct experiments on real-world datasets that corroborate our theoretical findings. We believe these insights are helpful in guiding the future design of domain adaptation and representation learning algorithms.

Compared with the last version, the current one adds a new corollary for the case of different feature transformations (encoders) on source/target domains. Fix a typo in Fig. 1

References in corpus (16)

• Learning Transferable Features with Deep Adaptation Networks
• Unsupervised Domain Adaptation with Residual Transfer Networks
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• Domain Separation Networks
• Adversarial Discriminative Domain Adaptation
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Domain-Adversarial Neural Networks
• Detecting and Correcting for Label Shift with Black Box Predictors
• Multi-Adversarial Domain Adaptation
• Multiple Source Adaptation and the Renyi Divergence
• Minimax Statistical Learning with Wasserstein Distances
• Augmented Cyclic Adversarial Learning for Low Resource Domain Adaptation
• Support and Invertibility in Domain-Invariant Representations
• Aspect-augmented Adversarial Networks for Domain Adaptation
• Efficient Multitask Feature and Relationship Learning

Cited by in corpus (81)

• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Generalizing to unseen domains via distribution matching
• DIVA: Domain Invariant Variational Autoencoders
• SCL: Towards Accurate Domain Adaptive Object Detection via Gradient Detach Based Stacked Complementary Losses
• In Search of Lost Domain Generalization
• Knowledge Transfer between Buildings for Seismic Damage Diagnosis through Adversarial Learning
• Domain Generalization using Causal Matching
• Domain Generalization via Optimal Transport with Metric Similarity Learning
• Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift
• Cycle Self-Training for Domain Adaptation
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
• Support and Invertibility in Domain-Invariant Representations
• Learning joint segmentation of tissues and brain lesions from task-specific hetero-modal domain-shifted datasets
• Rethinking Distributional Matching Based Domain Adaptation
• Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
• A Causal View on Robustness of Neural Networks
• Invariant Risk Minimization Games
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Inherent Tradeoffs in Learning Fair Representations
• Domain Aggregation Networks for Multi-Source Domain Adaptation
• Minimum Class Confusion for Versatile Domain Adaptation
• Robust Local Preserving and Global Aligning Network for Adversarial Domain Adaptation
• Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering
• Selecting Data Augmentation for Simulating Interventions
• Cluster Alignment with a Teacher for Unsupervised Domain Adaptation
• A survey on domain adaptation theory: learning bounds and theoretical guarantees
• Understanding Self-Training for Gradual Domain Adaptation
• Transferable Calibration with Lower Bias and Variance in Domain Adaptation
• Vicinal and categorical domain adaptation
• Desiderata for Representation Learning: A Causal Perspective
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners
• Domain adaptation under structural causal models
• Learning Overlapping Representations for the Estimation of Individualized Treatment Effects
• Minimax Lower Bounds for Transfer Learning with Linear and One-hidden Layer Neural Networks
• Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity
• Flow imaging as an alternative to pressure transducers through vision transformers and convolutional neural networks
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation
• Anomaly Detection with Domain Adaptation
• Fundamental Limits and Tradeoffs in Invariant Representation Learning
• Domain Invariant Representation Learning with Domain Density Transformations
• Improving Unsupervised Domain Adaptation with Variational Information Bottleneck
• Latent Causal Invariant Model
• A Theory of Label Propagation for Subpopulation Shift
• Domain Adaptation with Auxiliary Target Domain-Oriented Classifier
• Enforcing Predictive Invariance across Structured Biomedical Domains
• Curriculum Manager for Source Selection in Multi-Source Domain Adaptation
• Class-imbalanced Domain Adaptation: An Empirical Odyssey
• Model-based Policy Optimization with Unsupervised Model Adaptation
• Continuous Transfer Learning with Label-informed Distribution Alignment
• MTS-CycleGAN: An Adversarial-based Deep Mapping Learning Network for Multivariate Time Series Domain Adaptation Applied to the Ironmaking Industry
• Learning to Match Distributions for Domain Adaptation
• MLDemon: Deployment Monitoring for Machine Learning Systems
• Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning
• Adaptive Invariance for Molecule Property Prediction
• Cross-Domain Sentiment Classification with In-Domain Contrastive Learning
• Unsupervised Domain Adaptation with Progressive Domain Augmentation
• Gradual Domain Adaptation via Self-Training of Auxiliary Models
• An Information-theoretic Approach to Distribution Shifts
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer
• Optimization-based Causal Estimation from Heterogenous Environments
• Multi-task Learning by Leveraging the Semantic Information
• Mining Label Distribution Drift in Unsupervised Domain Adaptation
• Discriminative Active Learning for Domain Adaptation
• Representation Bayesian Risk Decompositions and Multi-Source Domain Adaptation
• Target Consistency for Domain Adaptation: when Robustness meets Transferability
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• A Label Proportions Estimation Technique for Adversarial Domain Adaptation in Text Classification
• Sim2Real for Self-Supervised Monocular Depth and Segmentation
• Interventional Domain Adaptation
• Conditional Bures Metric for Domain Adaptation
• Multi-Purposing Domain Adaptation Discriminators for Pseudo Labeling Confidence
• The Role of Embedding Complexity in Domain-invariant Representations
• Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain Adaptation using Structurally Regularized Deep Clustering
• Domain Adaptive Semantic Segmentation without Source Data
• KL Guided Domain Adaptation
• Cross-domain error minimization for unsupervised domain adaptation
• NI-UDA: Graph Adversarial Domain Adaptation from Non-shared-and-Imbalanced Big Data to Small Imbalanced Applications
• Task Guided Compositional Representation Learning for ZDA