When is invariance useful in an Out-of-Distribution Generalization problem ?

Abstract

The goal of Out-of-Distribution (OOD) generalization problem is to train a predictor that generalizes on all environments. Popular approaches in this field use the hypothesis that such a predictor shall be an \textit{invariant predictor} that captures the mechanism that remains constant across environments. While these approaches have been experimentally successful in various case studies, there is still much room for the theoretical validation of this hypothesis. This paper presents a new set of theoretical conditions necessary for an invariant predictor to achieve the OOD optimality. Our theory not only applies to non-linear cases, but also generalizes the necessary condition used in \citet{rojas2018invariant}. We also derive Inter Gradient Alignment algorithm from our theory and demonstrate its competitiveness on MNIST-derived benchmark datasets as well as on two of the three \textit{Invariance Unit Tests} proposed by \citet{aubinlinear}.

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The information bottleneck method
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Shortcut Learning in Deep Neural Networks
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• Invariant Risk Minimization
• On Learning Invariant Representation for Domain Adaptation
• Invariant Models for Causal Transfer Learning
• Causally Regularized Learning with Agnostic Data Selection Bias
• The Risks of Invariant Risk Minimization
• Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
• Learning explanations that are hard to vary
• Anchor regression: heterogeneous data meets causality
• Variational Information Maximization for Feature Selection
• Does Invariant Risk Minimization Capture Invariance?
• Domain-Invariant Adversarial Learning for Unsupervised Domain Adaption
• Linear unit-tests for invariance discovery
• Entropy-Constrained Maximizing Mutual Information Quantization
• Accounting for Unobserved Confounding in Domain Generalization

Cited by in corpus (13)

• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
• Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
• Linear unit-tests for invariance discovery
• Out-of-Distribution Generalization Analysis via Influence Function
• Quantifying and Improving Transferability in Domain Generalization
• SparCAssist: A Model Risk Assessment Assistant Based on Sparse Generated Counterfactuals
• Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
• Iterative VAE as a predictive brain model for out-of-distribution generalization
• A call for better unit testing for invariant risk minimisation
• On Invariance Penalties for Risk Minimization
• Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
• An Empirical Framework for Domain Generalization in Clinical Settings