Benchmarks for Deep Off-Policy Evaluation

Abstract

Off-policy evaluation (OPE) holds the promise of being able to leverage large, offline datasets for both evaluating and selecting complex policies for decision making. The ability to learn offline is particularly important in many real-world domains, such as in healthcare, recommender systems, or robotics, where online data collection is an expensive and potentially dangerous process. Being able to accurately evaluate and select high-performing policies without requiring online interaction could yield significant benefits in safety, time, and cost for these applications. While many OPE methods have been proposed in recent years, comparing results between papers is difficult because currently there is a lack of a comprehensive and unified benchmark, and measuring algorithmic progress has been challenging due to the lack of difficult evaluation tasks. In order to address this gap, we present a collection of policies that in conjunction with existing offline datasets can be used for benchmarking off-policy evaluation. Our tasks include a range of challenging high-dimensional continuous control problems, with wide selections of datasets and policies for performing policy selection. The goal of our benchmark is to provide a standardized measure of progress that is motivated from a set of principles designed to challenge and test the limits of existing OPE methods. We perform an evaluation of state-of-the-art algorithms and provide open-source access to our data and code to foster future research in this area.

ICLR 2021 paper. Policies and evaluation code are available at https://github.com/google-research/deep_ope

References in corpus (15)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• Critic Regularized Regression
• Acme: A Research Framework for Distributed Reinforcement Learning
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters
• Off-Policy Evaluation via the Regularized Lagrangian
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Batch Stationary Distribution Estimation

Cited by in corpus (8)

• A Minimalist Approach to Offline Reinforcement Learning
• What Matters in Learning from Offline Human Demonstrations for Robot Manipulation
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Active Offline Policy Selection