Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs

Abstract

Robust MDPs (RMDPs) can be used to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution are determined by the ambiguity set---the set of plausible transition probabilities---which is usually constructed as a multi-dimensional confidence region. Existing methods construct ambiguity sets as confidence regions using concentration inequalities which leads to overly conservative solutions. This paper proposes a new paradigm that can achieve better solutions with the same robustness guarantees without using confidence regions as ambiguity sets. To incorporate prior knowledge, our algorithms optimize the size and position of ambiguity sets using Bayesian inference. Our theoretical analysis shows the safety of the proposed method, and the empirical results demonstrate its practical promise.

References in corpus (2)

• Safe Policy Improvement by Minimizing Robust Baseline Regret
• Approximate Dynamic Programming By Minimizing Distributionally Robust Bounds

Cited by in corpus (9)

• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Byzantine-Resilient Decentralized TD Learning with Linear Function Approximation
• A Bayesian Approach to Robust Reinforcement Learning
• Distributional Robustness and Regularization in Reinforcement Learning
• Deep Bayesian Reward Learning from Preferences
• Explicit Explore, Exploit, or Escape ($E^4$): near-optimal safety-constrained reinforcement learning in polynomial time
• Towards Robust Off-Policy Evaluation via Human Inputs
• Optimizing Norm-Bounded Weighted Ambiguity Sets for Robust MDPs
• LiRA: Light-Robust Adversary for Model-based Reinforcement Learning in Real World