Active Offline Policy Selection

Abstract

This paper addresses the problem of policy selection in domains with abundant logged data, but with a restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and recommendation domains among others. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation. Yet, large amounts of online interactions are often not possible in practice. To overcome this problem, we introduce active offline policy selection - a novel sequential decision approach that combines logged data with online interaction to identify the best policy. We use OPE estimates to warm start the online evaluation. Then, in order to utilize the limited environment interactions wisely we decide which policy to evaluate next based on a Bayesian optimization method with a kernel that represents policy similarity. We use multiple benchmarks, including real-world robotics, with a large number of candidate policies to show that the proposed approach improves upon state-of-the-art OPE estimates and pure online policy evaluation.

arXiv commentPresented at NeurIPS 2021

References in corpus (33)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Playing Atari with Deep Reinforcement Learning
• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Conservative Q-Learning for Offline Reinforcement Learning
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Distributed Distributional Deterministic Policy Gradients
• Doubly Robust Policy Evaluation and Learning
• Challenges of Real-World Reinforcement Learning
• MOPO: Model-based Offline Policy Optimization
• Implicit Quantile Networks for Distributional Reinforcement Learning
• dm_control: Software and Tasks for Continuous Control
• MOReL : Model-Based Offline Reinforcement Learning
• Chip Placement with Deep Reinforcement Learning
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• Critic Regularized Regression
• Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
• Bayesian Optimization in AlphaGo
• Acme: A Research Framework for Distributed Reinforcement Learning
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• Hybrid Batch Bayesian Optimization
• Benchmarks for Deep Off-Policy Evaluation
• Off-Policy Evaluation via the Regularized Lagrangian
• Discovering General-Purpose Active Learning Strategies
• Policy Evaluation Networks
• Active Reinforcement Learning: Observing Rewards at a Cost
• Ready Policy One: World Building Through Active Learning
• Learning When-to-Treat Policies
• Offline Policy Selection under Uncertainty
• Reverb: A Framework For Experience Replay
• Regularized Behavior Value Estimation