When is invariance useful in an Out-of-Distribution Generalization problem ?
arXiv:2008.01883
Abstract
The goal of Out-of-Distribution (OOD) generalization problem is to train a predictor that generalizes on all environments. Popular approaches in this field use the hypothesis that such a predictor shall be an \textit{invariant predictor} that captures the mechanism that remains constant across environments. While these approaches have been experimentally successful in various case studies, there is still much room for the theoretical validation of this hypothesis. This paper presents a new set of theoretical conditions necessary for an invariant predictor to achieve the OOD optimality. Our theory not only applies to non-linear cases, but also generalizes the necessary condition used in \citet{rojas2018invariant}. We also derive Inter Gradient Alignment algorithm from our theory and demonstrate its competitiveness on MNIST-derived benchmark datasets as well as on two of the three \textit{Invariance Unit Tests} proposed by \citet{aubinlinear}.
References in corpus (8)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- On Learning Invariant Representation for Domain Adaptation
- The Risks of Invariant Risk Minimization
- Learning explanations that are hard to vary
- Does Invariant Risk Minimization Capture Invariance?
- Linear unit-tests for invariance discovery
- Entropy-Constrained Maximizing Mutual Information Quantization
Cited by in corpus (13)
- SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization
- Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?
- Empirical or Invariant Risk Minimization? A Sample Complexity Perspective
- Linear unit-tests for invariance discovery
- Out-of-Distribution Generalization Analysis via Influence Function
- Quantifying and Improving Transferability in Domain Generalization
- SparCAssist: A Model Risk Assessment Assistant Based on Sparse Generated Counterfactuals
- Linear Regression Games: Convergence Guarantees to Approximate Out-of-Distribution Solutions
- Iterative VAE as a predictive brain model for out-of-distribution generalization
- A call for better unit testing for invariant risk minimisation
- On Invariance Penalties for Risk Minimization
- Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
- An Empirical Framework for Domain Generalization in Clinical Settings