Detecting and Correcting for Label Shift with Black Box Predictors

Abstract

Faced with distribution shift between training and test set, we wish to detect and quantify the shift, and to correct our classifiers without test set labels. Motivated by medical diagnosis, where diseases (targets) cause symptoms (observations), we focus on label shift, where the label marginal $p(y)$ changes but the conditional $p(x|y)$ does not. We propose Black Box Shift Estimation (BBSE) to estimate the test distribution $p(y)$. BBSE exploits arbitrary black box predictors to reduce dimensionality prior to shift correction. While better predictors give tighter estimates, BBSE works even when predictors are biased, inaccurate, or uncalibrated, so long as their confusion matrices are invertible. We prove BBSE's consistency, bound its error, and introduce a statistical test that uses BBSE to detect shift. We also leverage BBSE to correct classifiers. Experiments demonstrate accurate estimates and improved prediction, even on high-dimensional datasets of natural images.

Published at the International Conference on Machine Learning (ICML) 2018

References in corpus (1)

• On Causal and Anticausal Learning

Cited by in corpus (28)

• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts
• On Learning Invariant Representation for Domain Adaptation
• Causality for Machine Learning
• Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
• Domain Adaptation with Asymmetrically-Relaxed Distribution Alignment
• Fairness without Demographics through Adversarially Reweighted Learning
• Robust Validation: Confident Predictions Even When Distributions Shift
• Support and Invertibility in Domain-Invariant Representations
• Monitoring and explainability of models in production
• Domain Adaptation as a Problem of Inference on Graphical Models
• Confidence intervals for class prevalences under prior probability shift
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Fairness Warnings and Fair-MAML: Learning Fairly with Minimal Data
• Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges
• Risk Variance Penalization
• Optimizing Data Usage via Differentiable Rewards
• Factorizable Joint Shift in Multinomial Classification
• Active Learning under Label Shift
• Multi-task Learning by Leveraging the Semantic Information
• Robust Classification under Class-Dependent Domain Shift
• A Label Proportions Estimation Technique for Adversarial Domain Adaptation in Text Classification
• Domain Adaptation in Highly Imbalanced and Overlapping Datasets
• The Influence of Faulty Labels in Data Sets on Human Pose Estimation
• Inference with generalizable classifier predictions
• Detection of Dataset Shifts in Learning-Enabled Cyber-Physical Systems using Variational Autoencoder for Regression
• Unsupervised Recalibration