Support and Invertibility in Domain-Invariant Representations

Abstract

Learning domain-invariant representations has become a popular approach to unsupervised domain adaptation and is often justified by invoking a particular suite of theoretical results. We argue that there are two significant flaws in such arguments. First, the results in question hold only for a fixed representation and do not account for information lost in non-invertible transformations. Second, domain invariance is often a far too strict requirement and does not always lead to consistent estimation, even under strong and favorable assumptions. In this work, we give generalization bounds for unsupervised domain adaptation that hold for any representation function by acknowledging the cost of non-invertibility. In addition, we show that penalizing distance between densities is often wasteful and propose a bound based on measuring the extent to which the support of the source domain covers the target domain. We perform experiments on well-known benchmarks that illustrate the short-comings of current standard practice.

References in corpus (9)

• Domain Separation Networks
• Domain Generalization via Invariant Feature Representation
• Domain Adaptation: Learning Bounds and Algorithms
• A General Algorithm for Deciding Transportability of Experimental Results
• On Learning Invariant Representation for Domain Adaptation
• Estimating individual treatment effect: generalization bounds and algorithms
• Deep Transfer Learning with Joint Adaptation Networks
• Detecting and Correcting for Label Shift with Black Box Predictors
• Overlap in Observational Studies with High-Dimensional Covariates

Cited by in corpus (25)

• Invariant Risk Minimization
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• On Learning Invariant Representation for Domain Adaptation
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift
• The Risks of Invariant Risk Minimization
• In Search of Lost Domain Generalization
• Invariant Risk Minimization Games
• Domain Aggregation Networks for Multi-Source Domain Adaptation
• A Prototype-Oriented Framework for Unsupervised Domain Adaptation
• Selecting Data Augmentation for Simulating Interventions
• Latent Causal Invariant Model
• SurvITE: Learning Heterogeneous Treatment Effects from Time-to-Event Data
• Learning to Match Distributions for Domain Adaptation
• Continuous Transfer Learning with Label-informed Distribution Alignment
• Learning Decomposed Representation for Counterfactual Inference
• Understanding the Limits of Unsupervised Domain Adaptation via Data Poisoning
• A Fine-Grained Analysis on Distribution Shift
• On Localized Discrepancy for Domain Adaptation
• Factorizable Joint Shift in Multinomial Classification
• Regret Minimization for Causal Inference on Large Treatment Space
• KL Guided Domain Adaptation
• The Role of Embedding Complexity in Domain-invariant Representations
• Task Guided Compositional Representation Learning for ZDA
• $\beta$-Intact-VAE: Identifying and Estimating Causal Effects under Limited Overlap