A Practical Guide of Off-Policy Evaluation for Bandit Problems

Abstract

Off-policy evaluation (OPE) is the problem of estimating the value of a target policy from samples obtained via different policies. Recently, applying OPE methods for bandit problems has garnered attention. For the theoretical guarantees of an estimator of the policy value, the OPE methods require various conditions on the target policy and policy used for generating the samples. However, existing studies did not carefully discuss the practical situation where such conditions hold, and the gap between them remains. This paper aims to show new results for bridging the gap. Based on the properties of the evaluation policy, we categorize OPE situations. Then, among practical applications, we mainly discuss the best policy selection. For the situation, we propose a meta-algorithm based on existing OPE estimators. We investigate the proposed concepts using synthetic and open real-world datasets in experiments.

References in corpus (7)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Batched bandit problems
• Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models
• More Efficient Off-Policy Evaluation through Regularized Targeted Learning
• Off-Policy Evaluation and Learning for External Validity under a Covariate Shift
• Confidence Interval for Off-Policy Evaluation from Dependent Samples via Bandit Algorithm: Approach from Standardized Martingales

Cited by in corpus (3)

• Quality Metrics in Recommender Systems: Do We Calculate Metrics Consistently?
• Probabilistic design of optimal sequential decision-making algorithms in learning and control
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation