On Invariance Penalties for Risk Minimization

Abstract

The Invariant Risk Minimization (IRM) principle was first proposed by Arjovsky et al. [2019] to address the domain generalization problem by leveraging data heterogeneity from differing experimental conditions. Specifically, IRM seeks to find a data representation under which an optimal classifier remains invariant across all domains. Despite the conceptual appeal of IRM, the effectiveness of the originally proposed invariance penalty has recently been brought into question. In particular, there exists counterexamples for which that invariance penalty can be arbitrarily small for non-invariant data representations. We propose an alternative invariance penalty by revisiting the Gramian matrix of the data representation. We discuss the role of its eigenvalues in the relationship between the risk and the invariance penalty, and demonstrate that it is ill-conditioned for said counterexamples. The proposed approach is guaranteed to recover an invariant representation for linear settings under mild non-degeneracy conditions. Its effectiveness is substantiated by experiments on DomainBed and InvarianceUnitTest, two extensive test beds for domain generalization.

References in corpus (11)

• Invariant Risk Minimization
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• In Search of Lost Domain Generalization
• Learning explanations that are hard to vary
• Out of Distribution Generalization in Machine Learning
• Unshuffling Data for Improved Generalization
• When is invariance useful in an Out-of-Distribution Generalization problem ?
• Does Invariant Risk Minimization Capture Invariance?
• Learning Causal Models Online
• Linear unit-tests for invariance discovery
• Invariant Representation Learning for Treatment Effect Estimation