Does Invariant Risk Minimization Capture Invariance?

Abstract

We show that the Invariant Risk Minimization (IRM) formulation of Arjovsky et al. (2019) can fail to capture "natural" invariances, at least when used in its practical "linear" form, and even on very simple problems which directly follow the motivating examples for IRM. This can lead to worse generalization on new environments, even when compared to unconstrained ERM. The issue stems from a significant gap between the linear variant (as in their concrete method IRMv1) and the full non-linear IRM formulation. Additionally, even when capturing the "right" invariances, we show that it is possible for IRM to learn a sub-optimal predictor, due to the loss function not being invariant across environments. The issues arise even when measuring invariance on the population distributions, but are exacerbated by the fact that IRM is extremely fragile to sampling.

Code is available in the arXiv ancillary files, linked from this page

References in corpus (4)

• Invariant Risk Minimization
• Invariant Models for Causal Transfer Learning
• The Risks of Invariant Risk Minimization
• A survey on domain adaptation theory: learning bounds and theoretical guarantees

Cited by in corpus (17)

• Towards Out-Of-Distribution Generalization: A Survey
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Environment Inference for Invariant Learning
• Generalizing Graph Neural Networks on Out-Of-Distribution Graphs
• Model-Based Domain Generalization
• Meta-Learned Invariant Risk Minimization
• Iterative Feature Matching: Toward Provable Domain Generalization with Logarithmic Environments
• Towards Principled Disentanglement for Domain Generalization
• Optimization-based Causal Estimation from Heterogenous Environments
• On Invariance Penalties for Risk Minimization
• A call for better unit testing for invariant risk minimisation
• Regularizing towards Causal Invariance: Linear Models with Proxies
• An Empirical Framework for Domain Generalization in Clinical Settings
• Task Guided Compositional Representation Learning for ZDA
• Counterfactual Supervision-based Information Bottleneck for Out-of-Distribution Generalization