Risk Variance Penalization
arXiv:2006.07544
Abstract
The key of the out-of-distribution (OOD) generalization is to generalize invariance from training domains to target domains. The variance risk extrapolation (V-REx) is a practical OOD method, which depends on a domain-level regularization but lacks theoretical verifications about its motivation and utility. This article provides theoretical insights into V-REx by studying a variance-based regularizer. We propose Risk Variance Penalization (RVP), which slightly changes the regularization of V-REx but addresses the theory concerns about V-REx. We provide theoretical explanations and a theory-inspired tuning scheme for the regularization parameter of RVP. Our results point out that RVP discovers a robust predictor. Finally, we experimentally show that the proposed regularizer can find an invariant predictor under certain conditions.
References in corpus (7)
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- Distributionally Robust Logistic Regression
- Invariant Models for Causal Transfer Learning
- Detecting and Correcting for Label Shift with Black Box Predictors
- Reliable Decision Support using Counterfactual Models
- Domain Adaptation by Using Causal Inference to Predict Invariant Conditional Distributions
- Causal Inference and Mechanism Clustering of A Mixture of Additive Noise Models