Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport

Abstract

Classical supervised learning produces unreliable models when training and target distributions differ, with most existing solutions requiring samples from the target domain. We propose a proactive approach which learns a relationship in the training domain that will generalize to the target domain by incorporating prior knowledge of aspects of the data generating process that are expected to differ as expressed in a causal selection diagram. Specifically, we remove variables generated by unstable mechanisms from the joint factorization to yield the Surgery Estimator---an interventional distribution that is invariant to the differences across environments. We prove that the surgery estimator finds stable relationships in strictly more scenarios than previous approaches which only consider conditional relationships, and demonstrate this in simulated experiments. We also evaluate on real world data for which the true causal diagram is unknown, performing competitively against entirely data-driven approaches.

arXiv commentIn Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), 2019. Previously presented at the NeurIPS 2018 Causal Learning Workshop

References in corpus (10)

• Confounding variables can degrade generalization performance of radiological deep learning models
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• Invariant Models for Causal Transfer Learning
• On Causal and Anticausal Learning
• Reliable Decision Support using Counterfactual Models
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Nested Markov Properties for Acyclic Directed Mixed Graphs
• Anchor regression: heterogeneous data meets causality
• Tutorial: Safe and Reliable Machine Learning
• Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms

Cited by in corpus (4)

• Causality for Machine Learning
• Controlling for Confounders in Multimodal Emotion Classification via Adversarial Learning
• Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings
• AI for Anticipatory Action: Moving Beyond Climate Forecasting