Safe Policy Improvement by Minimizing Robust Baseline Regret

Abstract

An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, i.e., a policy that is guaranteed to perform at least as well as a given baseline strategy. In this paper, we develop and analyze a new model-based approach to compute a safe policy when we have access to an inaccurate dynamics model of the system with known accuracy guarantees. Our proposed robust method uses this (inaccurate) model to directly minimize the (negative) regret w.r.t. the baseline policy. Contrary to the existing approaches, minimizing the regret allows one to improve the baseline policy in states with accurate dynamics and seamlessly fall back to the baseline policy, otherwise. We show that our formulation is NP-hard and propose an approximate algorithm. Our empirical results on several domains show that even this relatively simple approximate algorithm can significantly outperform standard approaches.

Cited by in corpus (31)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Conservative Q-Learning for Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• MOReL : Model-Based Offline Reinforcement Learning
• AI Safety Gridworlds
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• COMBO: Conservative Offline Model-Based Policy Optimization
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Confounding-Robust Policy Improvement
• The Importance of Pessimism in Fixed-Dataset Policy Optimization
• Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
• Policy Certificates: Towards Accountable Reinforcement Learning
• Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning
• On Ensuring that Intelligent Machines Are Well-Behaved
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control
• Partial Policy Iteration for L1-Robust Markov Decision Processes
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
• Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
• Predictive Coding for Locally-Linear Control
• Safe Exploration for Identifying Linear Systems via Robust Optimization
• Online Safety Assurance for Deep Reinforcement Learning
• Tight Bayesian Ambiguity Sets for Robust MDPs
• Optimizing Norm-Bounded Weighted Ambiguity Sets for Robust MDPs
• Configurable Markov Decision Processes
• Bayesian Robust Optimization for Imitation Learning
• Defining Admissible Rewards for High Confidence Policy Evaluation
• Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces
• Greedy UnMixing for Q-Learning in Multi-Agent Reinforcement Learning
• A Short Survey on Probabilistic Reinforcement Learning
• Entropic Risk Constrained Soft-Robust Policy Optimization
• Improved Algorithms for Conservative Exploration in Bandits