The Risks of Invariant Risk Minimization

Abstract

Invariant Causal Prediction (Peters et al., 2016) is a technique for out-of-distribution generalization which assumes that some aspects of the data distribution vary across the training set but that the underlying causal mechanisms remain constant. Recently, Arjovsky et al. (2019) proposed Invariant Risk Minimization (IRM), an objective based on this idea for learning deep, invariant features of data which are a complex function of latent variables; many alternatives have subsequently been suggested. However, formal guarantees for all of these works are severely lacking. In this paper, we present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model. In the linear case, we show simple conditions under which the optimal solution succeeds or, more often, fails to recover the optimal invariant predictor. We furthermore present the very first results in the non-linear regime: we demonstrate that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve. Thus, in this setting we find that IRM and its alternatives fundamentally do not improve over standard Empirical Risk Minimization.

arXiv commentICLR 2021 Camera-Ready

References in corpus (12)

• Conditional Adversarial Domain Adaptation
• Domain Generalization via Invariant Feature Representation
• Adversarially Robust Generalization Requires More Data
• Do ImageNet Classifiers Generalize to ImageNet?
• Estimating high-dimensional intervention effects from observational data
• Invariant Risk Minimization
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Invariant Models for Causal Transfer Learning
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Domain Generalization using Causal Matching
• Support and Invertibility in Domain-Invariant Representations
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization

Cited by in corpus (32)

• Towards Out-Of-Distribution Generalization: A Survey
• Causal Attention for Interpretable and Generalizable Graph Classification
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Graph Rationalization with Environment-based Augmentations
• Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Nonlinear Invariant Risk Minimization: A Causal Approach
• Does Invariant Risk Minimization Capture Invariance?
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Out-of-distribution Prediction with Invariant Risk Minimization: The Limitation and An Effective Fix
• Invariant Representation Learning for Treatment Effect Estimation
• On Calibration and Out-of-domain Generalization
• Environment Inference for Invariant Learning
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• Size-Invariant Graph Representations for Graph Classification Extrapolations
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Model-Based Domain Generalization
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization
• Towards Principled Disentanglement for Domain Generalization
• Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments
• Meta-Learned Invariant Risk Minimization
• Model-Invariant State Abstractions for Model-Based Reinforcement Learning
• An Information-theoretic Approach to Distribution Shifts
• Optimization-based Causal Estimation from Heterogenous Environments
• Neural Networks for Learning Counterfactual G-Invariances from Single Environments
• Causal Learning for Socially Responsible AI
• Regularizing towards Causal Invariance: Linear Models with Proxies
• Learning Under Adversarial and Interventional Shifts
• Domain Conditional Predictors for Domain Adaptation
• An Empirical Framework for Domain Generalization in Clinical Settings
• Invariant Risk Minimisation for Cross-Organism Inference: Substituting Mouse Data for Human Data in Human Risk Factor Discovery