Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning

Abstract

We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.

References in corpus (7)

• Doubly Robust Policy Evaluation and Learning
• Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processes
• Information-Theoretic Considerations in Batch Reinforcement Learning
• Off-Policy Policy Gradient with State Distribution Correction
• Benchmarks for Deep Off-Policy Evaluation
• CoinDICE: Off-Policy Confidence Interval Estimation
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization

Cited by in corpus (29)

• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
• Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems
• Evaluating the Robustness of Off-Policy Evaluation
• NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Benchmarks for Deep Off-Policy Evaluation
• Learning Deep Features in Instrumental Variable Regression
• Offline RL Without Off-Policy Evaluation
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Games
• Bootstrapping Fitted Q-Evaluation for Off-Policy Inference
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• Adaptive Estimator Selection for Off-Policy Evaluation
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Safe Autonomous Racing via Approximate Reachability on Ego-vision
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulation
• Off-Policy Evaluation in Partially Observed Markov Decision Processes under Sequential Ignorability
• On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networks
• Minimax Model Learning
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Active Offline Policy Selection
• On Convergence of Gradient Expected Sarsa($\lambda$)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach
• Off-Policy Optimization of Portfolio Allocation Policies under Constraints