Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms

Abstract

Predictive models can fail to generalize from training to deployment environments because of dataset shift, posing a threat to model reliability and the safety of downstream decisions made in practice. Instead of using samples from the target distribution to reactively correct dataset shift, we use graphical knowledge of the causal mechanisms relating variables in a prediction problem to proactively remove relationships that do not generalize across environments, even when these relationships may depend on unobserved variables (violations of the "no unobserved confounders" assumption). To accomplish this, we identify variables with unstable paths of statistical influence and remove them from the model. We also augment the causal graph with latent counterfactual variables that isolate unstable paths of statistical influence, allowing us to retain stable paths that would otherwise be removed. Our experiments demonstrate that models that remove vulnerable variables and use estimates of the latent variables transfer better, often outperforming in the target domain despite some accuracy loss in the training domain.

arXiv commentProceedings of the 34th Conference on Uncertainty in Artificial Intelligence (UAI), 2018. Revised from print version

References in corpus (8)

• Certifying Some Distributional Robustness with Principled Adversarial Training
• Causal Inference in the Presence of Latent Variables and Selection Bias
• An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection
• Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier
• Anchor regression: heterogeneous data meets causality
• What Counterfactuals Can Be Tested
• Bayesian Nonparametric Causal Inference: Information Rates and Learning Algorithms
• Learning (Predictive) Risk Scores in the Presence of Censoring due to Interventions

Cited by in corpus (7)

• Towards Out-Of-Distribution Generalization: A Survey
• Evaluating Model Robustness and Stability to Dataset Shift
• Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport
• Robust Learning with the Hilbert-Schmidt Independence Criterion
• Counterfactual Predictions under Runtime Confounding
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
• Stable predictions for health related anticausal prediction tasks affected by selection biases: the need to deconfound the test set features