Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning

Abstract

In this paper we present a new way of predicting the performance of a reinforcement learning policy given historical data that may have been generated by a different policy. The ability to evaluate a policy from historical data is important for applications where the deployment of a bad policy can be dangerous or costly. We show empirically that our algorithm produces estimates that often have orders of magnitude lower mean squared error than existing methods---it makes more efficient use of the available data. Our new estimator is based on two advances: an extension of the doubly robust estimator (Jiang and Li, 2015), and a new way to mix between model based estimates and importance sampling based estimates.

References in corpus (4)

• Doubly Robust Policy Evaluation and Learning
• Doubly Robust Off-policy Value Evaluation for Reinforcement Learning
• Emphatic Temporal-Difference Learning
• A Notation for Markov Decision Processes

Cited by in corpus (7)

• Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• Designing Reinforcement Learning Algorithms for Digital Interventions: Pre-implementation Guidelines
• Multi-step Off-policy Learning Without Importance Sampling Ratios
• Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
• A comparative study of counterfactual estimators
• Semi-Supervised Off Policy Reinforcement Learning
• Finding the Optimal Dynamic Treatment Regime Using Smooth Fisher Consistent Surrogate Loss