Invariant Models for Causal Transfer Learning
arXiv:1507.05333
Abstract
Methods of transfer learning try to combine knowledge from several related tasks (or domains) to improve performance on a test task. Inspired by causal methodology, we relax the usual covariate shift assumption and assume that it holds true for a subset of predictor variables: the conditional distribution of the target variable given this subset of predictors is invariant over all tasks. We show how this assumption can be motivated from ideas in the field of causality. We focus on the problem of Domain Generalization, in which no examples from the test task are observed. We prove that in an adversarial setting using this subset for prediction is optimal in Domain Generalization; we further provide examples, in which the tasks are sufficiently diverse and the estimator therefore outperforms pooling the data, even on average. If examples from the test task are available, we also provide a method to transfer knowledge from the training tasks and exploit all available features for prediction. However, we provide no guarantees for this method. We introduce a practical method which allows for automatic inference of the above subset and provide corresponding code. We present results on synthetic data sets and a gene deletion data set.
References in corpus (4)
Cited by in corpus (72)
- A review of domain adaptation without target labels
- Invariant Risk Minimization
- Towards Out-Of-Distribution Generalization: A Survey
- A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms
- Causality for Machine Learning
- How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
- The Risks of Invariant Risk Minimization
- In Search of Lost Domain Generalization
- Domain Generalization using Causal Matching
- Learning explanations that are hard to vary
- Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
- Learning Causal Semantic Representation for Out-of-Distribution Prediction
- Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
- Learning Neural Causal Models from Unknown Interventions
- Towards a Theoretical Framework of Out-of-Distribution Generalization
- Unshuffling Data for Improved Generalization
- Few-shot Domain Adaptation by Causal Mechanism Transfer
- When is invariance useful in an Out-of-Distribution Generalization problem ?
- Causality-based Feature Selection: Methods and Evaluations
- CASTLE: Regularization via Auxiliary Causal Graph Discovery
- Optimal Decision Making Under Strategic Behavior
- Nonlinear Invariant Risk Minimization: A Causal Approach
- Evaluating Model Robustness and Stability to Dataset Shift
- Does Invariant Risk Minimization Capture Invariance?
- Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness
- Selecting Data Augmentation for Simulating Interventions
- Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
- Domain adaptation under structural causal models
- Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport
- Invariant Representation Learning for Treatment Effect Estimation
- On the Transfer of Disentangled Representations in Realistic Settings
- Out-of-Distribution Generalization Analysis via Influence Function
- Latent Causal Invariant Model
- Distributional robustness of K-class estimators and the PULSE
- Distributional Anchor Regression
- Deep causal representation learning for unsupervised domain adaptation
- Explaining The Efficacy of Counterfactually Augmented Data
- Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests
- Searching for consistent associations with a multi-environment knockoff filter
- I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models
- Causality-aware counterfactual confounding adjustment for feature representations learned by deep models
- Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
- Environment Invariant Linear Least Squares
- Risk Variance Penalization
- Stable Prediction with Model Misspecification and Agnostic Distribution Shift
- In Search of Robust Measures of Generalization
- Learning Representations that Support Robust Transfer of Predictors
- Towards Principled Disentanglement for Domain Generalization
- Stable Prediction via Leveraging Seed Variable
- Structural Regularization
- Improving Model Robustness Using Causal Knowledge
- From Predictions to Decisions: Using Lookahead Regularization
- Identifying Invariant Factors Across Multiple Environments with KL Regression
- Causal aggregation: estimation and inference of causal effects by constraint-based data fusion
- Deconfounding and Causal Regularization for Stability and External Validity
- Optimization-based Causal Estimation from Heterogenous Environments
- Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
- Independent mechanism analysis, a new concept?
- Robust Generalization despite Distribution Shift via Minimum Discriminating Information
- Regularizing towards Causal Invariance: Linear Models with Proxies
- Learning Under Adversarial and Interventional Shifts
- Neural Networks for Learning Counterfactual G-Invariances from Single Environments
- Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization
- Inference with generalizable classifier predictions
- Causality and Generalizability: Identifiability and Learning Methods
- Distributionally Robust Learning with Stable Adversarial Training
- Robust Learning in Heterogeneous Contexts
- Balance-Subsampled Stable Prediction
- Adaptive Multi-Source Causal Inference
- Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
- Causal Transfer Random Forest: Combining Logged Data and Randomized Experiments for Robust Prediction
- Provable Guarantees on the Robustness of Decision Rules to Causal Interventions