Anchor regression: heterogeneous data meets causality

Abstract

We consider the problem of predicting a response variable from a set of covariates on a data set that differs in distribution from the training data. Causal parameters are optimal in terms of predictive accuracy if in the new distribution either many variables are affected by interventions or only some variables are affected, but the perturbations are strong. If the training and test distributions differ by a shift, causal parameters might be too conservative to perform well on the above task. This motivates anchor regression, a method that makes use of exogeneous variables to solve a relaxation of the causal minimax problem by considering a modification of the least-squares loss. The procedure naturally provides an interpolation between the solutions of ordinary least squares and two-stage least squares. We prove that the estimator satisfies predictive guarantees in terms of distributional robustness against shifts in a linear class; these guarantees are valid even if the instrumental variables assumptions are violated. If anchor regression and least squares provide the same answer (anchor stability), we establish that OLS parameters are invariant under certain distributional changes. Anchor regression is shown empirically to improve replicability and protect against distributional shifts.

References in corpus (16)

• Rejoinder to "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Pathwise coordinate optimization
• Certifying Some Distributional Robustness with Principled Adversarial Training
• Veridical Data Science
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Causal Discovery from Changes
• Learning stable and predictive structures in kinetic systems: Benefits of a causal approach

Cited by in corpus (37)

• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Towards Out-Of-Distribution Generalization: A Survey
• A causal view of compositional zero-shot recognition
• Out of Distribution Generalization in Machine Learning
• Learning Causal Semantic Representation for Out-of-Distribution Prediction
• Causality in cognitive neuroscience: concepts, challenges, and distributional robustness
• A Causal View on Robustness of Neural Networks
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• CASTLE: Regularization via Auxiliary Causal Graph Discovery
• Domain adaptation under structural causal models
• Counterfactual Normalization: Proactively Addressing Dataset Shift and Improving Reliability Using Causal Mechanisms
• Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport
• Robust Learning with the Hilbert-Schmidt Independence Criterion
• On Calibration and Out-of-domain Generalization
• Distributional robustness of K-class estimators and the PULSE
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Distributional Anchor Regression
• I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models
• Accounting for Unobserved Confounding in Domain Generalization
• Regularizing Double Machine Learning in Partially Linear Endogenous Models
• Active Invariant Causal Prediction: Experiment Selection through Stability
• Switching Regression Models and Causal Inference in the Presence of Discrete Latent Variables
• In Search of Robust Measures of Generalization
• A direct approach to detection and attribution of climate change
• Identifying Invariant Factors Across Multiple Environments with KL Regression
• From Predictions to Decisions: Using Lookahead Regularization
• Perturbation graphs, invariant prediction and causal relations in psychology
• Improving Model Robustness Using Causal Knowledge
• Model selection for estimation of causal parameters
• Towards Robust Off-Policy Evaluation via Human Inputs
• Deconfounding and Causal Regularization for Stability and External Validity
• Learning Under Adversarial and Interventional Shifts
• Independent mechanism analysis, a new concept?
• Training Data Subset Selection for Regression with Controlled Generalization Error
• Orthogonal Structure Search for Efficient Causal Discovery from Observational Data
• An Empirical Framework for Domain Generalization in Clinical Settings