Safe Policy Improvement with Baseline Bootstrapping

Abstract

This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Bootstrapping (SPIBB), is inspired by the knows-what-it-knows paradigm: it bootstraps the trained policy with the baseline when the uncertainty is high. Our first algorithm, ${\Pi }_b$-SPIBB, comes with SPI theoretical guarantees. We also implement a variant, ${\Pi }_{\le b}$-SPIBB, that is even more efficient in practice. We apply our algorithms to a motivational stochastic gridworld domain and further demonstrate on randomly generated MDPs the superiority of SPIBB with respect to existing algorithms, not only in safety but also in mean performance. Finally, we implement a model-free version of SPIBB and show its benefits on a navigation task with deep RL implementation called SPIBB-DQN, which is, to the best of our knowledge, the first RL algorithm relying on a neural network representation able to train efficiently and reliably from batch data, without any interaction with the environment.

accepted as a long oral at ICML2019

Cited by in corpus (41)

• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• An Optimistic Perspective on Offline Reinforcement Learning
• Location-routing Optimisation for Urban Logistics Using Mobile Parcel Locker Based on Hybrid Q-Learning Algorithm
• Route Planning for Last-Mile Deliveries Using Mobile Parcel Lockers: A Hybrid Q-Learning Network Approach
• COMBO: Conservative Offline Model-Based Policy Optimization
• What are the Statistical Limits of Offline RL with Linear Function Approximation?
• Provably Good Batch Reinforcement Learning Without Great Exploration
• Is Pessimism Provably Efficient for Offline RL?
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
• CoinDICE: Off-Policy Confidence Interval Estimation
• Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning
• Offline RL Without Off-Policy Evaluation
• Bellman-consistent Pessimism for Offline Reinforcement Learning
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Causality and Batch Reinforcement Learning: Complementary Approaches To Planning In Unknown Domains
• Offline Reinforcement Learning with Soft Behavior Regularization
• Budgeted Reinforcement Learning in Continuous State Space
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Conservative Exploration in Reinforcement Learning
• Corruption-Robust Offline Reinforcement Learning
• Optimizing Medical Treatment for Sepsis in Intensive Care: from Reinforcement Learning to Pre-Trial Evaluation
• Offline Reinforcement Learning with Pseudometric Learning
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Optimizing Percentile Criterion Using Robust MDPs
• Safe Policy Improvement Approaches on Discrete Markov Decision Processes
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
• Bayesian Robust Optimization for Imitation Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Safety Verification of Model Based Reinforcement Learning Controllers
• Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates
• Offline Contextual Bandits with Overparameterized Models
• Quantile Filtered Imitation Learning