Estimating Generalization under Distribution Shifts via Domain-Invariant Representations

Abstract

When machine learning models are deployed on a test distribution different from the training distribution, they can perform poorly, but overestimate their performance. In this work, we aim to better estimate a model's performance under distribution shift, without supervision. To do so, we use a set of domain-invariant predictors as a proxy for the unknown, true target labels. Since the error of the resulting risk estimate depends on the target risk of the proxy model, we study generalization of domain-invariant representations and show that the complexity of the latent representation has a significant influence on the target risk. Empirically, our approach (1) enables self-tuning of domain adaptation models, and (2) accurately estimates the target error of given models under distribution shift. Other applications include model selection, deciding early stopping and error detection.

arXiv admin note: text overlap with arXiv:1910.05804

References in corpus (7)

• Learning Transferable Features with Deep Adaptation Networks
• Conditional Adversarial Domain Adaptation
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Domain Adaptation: Learning Bounds and Algorithms
• A DIRT-T Approach to Unsupervised Domain Adaptation
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Minimax Statistical Learning with Wasserstein Distances

Cited by in corpus (5)

• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
• Towards Evaluating the Robustness of Neural Networks Learned by Transduction
• Distance Matters For Improving Performance Estimation Under Covariate Shift