An Information-theoretic Approach to Distribution Shifts
arXiv:2106.03783
Abstract
Safely deploying machine learning models to the real world is often a challenging process. Models trained with data obtained from a specific geographic location tend to fail when queried with data obtained elsewhere, agents trained in a simulation can struggle to adapt when deployed in the real world or novel environments, and neural networks that are fit to a subset of the population might carry some selection bias into their decision process. In this work, we describe the problem of data shift from a novel information-theoretic perspective by (i) identifying and describing the different sources of error, (ii) comparing some of the most promising objectives explored in the recent domain generalization, and fair classification literature. From our theoretical analysis and empirical evaluation, we conclude that the model selection procedure needs to be guided by careful considerations regarding the observed data, the factors used for correction, and the structure of the data-generating process.
References in corpus (22)
- Deep Visual Domain Adaptation: A Survey
- Equality of Opportunity in Supervised Learning
- Confounding variables can degrade generalization performance of radiological deep learning models
- Domain Generalization via Invariant Feature Representation
- Domain Adaptation: Learning Bounds and Algorithms
- Invariant Risk Minimization
- -divergence Inequalities
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Out-of-Distribution Generalization via Risk Extrapolation (REx)
- On Learning Invariant Representation for Domain Adaptation
- On Causal and Anticausal Learning
- Invariant Representations without Adversarial Training
- The Risks of Invariant Risk Minimization
- Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
- In Search of Lost Domain Generalization
- On Pinsker's Type Inequalities and Csiszar's f-divergences. Part I: Second and Fourth-Order Inequalities
- Domain Generalization using Causal Matching
- Learning explanations that are hard to vary
- Controllable Invariance through Adversarial Feature Learning
- A statistical framework for fair predictive algorithms
- Fairness Through Awareness
- BREEDS: Benchmarks for Subpopulation Shift