More Efficient Off-Policy Evaluation through Regularized Targeted Learning

Abstract

We study the problem of off-policy evaluation (OPE) in Reinforcement Learning (RL), where the aim is to estimate the performance of a new policy given historical data that may have been generated by a different policy, or policies. In particular, we introduce a novel doubly-robust estimator for the OPE problem in RL, based on the Targeted Maximum Likelihood Estimation principle from the statistical causal inference literature. We also introduce several variance reduction techniques that lead to impressive performance gains in off-policy evaluation. We show empirically that our estimator uniformly wins over existing off-policy evaluation methods across multiple RL environments and various levels of model misspecification. Finally, we further the existing theoretical analysis of estimators for the RL off-policy estimation problem by showing their $O_P(1/\sqrt{n})$ rate of convergence and characterizing their asymptotic distribution.

We are uploading the full paper with the appendix as of 12/12/2019, as we noticed that, unlike the main text, the appendix has not been made available on PMLR's website. The version of the appendix in this document is the same that we have been sending by email since June 2019 to readers who solicited it

Cited by in corpus (9)

• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency
• Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation
• A Practical Guide of Off-Policy Evaluation for Bandit Problems
• Debiasing Samples from Online Learning Using Bootstrap
• Control Variates for Slate Off-Policy Evaluation
• Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies
• Optimal Mixture Weights for Off-Policy Evaluation with Multiple Behavior Policies