Off-Policy Evaluation and Learning for External Validity under a Covariate Shift

Abstract

We consider evaluating and training a new policy for the evaluation data by using the historical data obtained from a different policy. The goal of off-policy evaluation (OPE) is to estimate the expected reward of a new policy over the evaluation data, and that of off-policy learning (OPL) is to find a new policy that maximizes the expected reward over the evaluation data. Although the standard OPE and OPL assume the same distribution of covariate between the historical and evaluation data, a covariate shift often exists, i.e., the distribution of the covariate of the historical data is different from that of the evaluation data. In this paper, we derive the efficiency bound of OPE under a covariate shift. Then, we propose doubly robust and efficient estimators for OPE and OPL under a covariate shift by using a nonparametric estimator of the density ratio between the historical and evaluation data distributions. We also discuss other possible estimators and compare their theoretical properties. Finally, we confirm the effectiveness of the proposed estimators through experiments.

References in corpus (10)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Doubly Robust Policy Evaluation and Learning
• Learning Representations for Counterfactual Inference
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
• Learning Weighted Representations for Generalization Across Designs
• Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
• The Offset Tree for Learning with Partial Labels
• Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
• Balanced off-policy evaluation in general action spaces

Cited by in corpus (8)

• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• A Practical Guide of Off-Policy Evaluation for Bandit Problems
• Off-Policy Evaluation of Bandit Algorithm from Dependent Samples under Batch Update Policy
• Off-Policy Exploitability-Evaluation in Two-Player Zero-Sum Markov Games
• Generalizing Off-Policy Learning under Sample Selection Bias
• Learning Under Adversarial and Interventional Shifts
• Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support
• The Adaptive Doubly Robust Estimator for Policy Evaluation in Adaptive Experiments and a Paradox Concerning Logging Policy