A Bayesian Approach to Robust Reinforcement Learning
arXiv:1905.08188
Abstract
Robust Markov Decision Processes (RMDPs) intend to ensure robustness with respect to changing or adversarial system behavior. In this framework, transitions are modeled as arbitrary elements of a known and properly structured uncertainty set and a robust optimal policy can be derived under the worst-case scenario. In this study, we address the issue of learning in RMDPs using a Bayesian approach. We introduce the Uncertainty Robust Bellman Equation (URBE) which encourages safe exploration for adapting the uncertainty set to new observations while preserving robustness. We propose a URBE-based algorithm, DQN-URBE, that scales this method to higher dimensional domains. Our experiments show that the derived URBE-based strategy leads to a better trade-off between less conservative solutions and robustness in the presence of model misspecification. In addition, we show that the DQN-URBE algorithm can adapt significantly faster to changing dynamics online compared to existing robust techniques with fixed uncertainty sets.
Accepted to UAI 2019
References in corpus (8)
- Reward Constrained Policy Optimization
- Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach
- Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty
- Soft-Robust Actor-Critic Policy-Gradient
- Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs
- Deep Robust Kalman Filter
- Learning Robust Options
- Tight Bayesian Ambiguity Sets for Robust MDPs
Cited by in corpus (5)
- An empirical investigation of the challenges of real-world reinforcement learning
- Partial Policy Iteration for L1-Robust Markov Decision Processes
- Risk-Averse Bayes-Adaptive Reinforcement Learning
- Robust Stochastic Bayesian Games for Behavior Space Coverage
- Survival of the Fittest: Evolutionary Adaptation of Policies for Environmental Shifts