Scaling Up Robust MDPs by Reinforcement Learning

Abstract

We consider large-scale Markov decision processes (MDPs) with parameter uncertainty, under the robust MDP paradigm. Previous studies showed that robust MDPs, based on a minimax approach to handle uncertainty, can be solved using dynamic programming for small to medium sized problems. However, due to the "curse of dimensionality", MDPs that model real-life problems are typically prohibitively large for such approaches. In this work we employ a reinforcement learning approach to tackle this planning problem: we develop a robust approximate dynamic programming method based on a projected fixed point equation to approximately solve large scale robust MDPs. We show that the proposed method provably succeeds under certain technical conditions, and demonstrate its effectiveness through simulation of an option pricing problem. To the best of our knowledge, this is the first attempt to scale up the robust MDPs paradigm.

Cited by in corpus (9)

• Action Robust Reinforcement Learning and Applications in Continuous Control
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• CoinDICE: Off-Policy Confidence Interval Estimation
• Constrained Model-based Reinforcement Learning with Robust Cross-Entropy Method
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Formal Language Constraints for Markov Decision Processes
• Pessimism About Unknown Unknowns Inspires Conservatism
• Learning to Design Games: Strategic Environments in Reinforcement Learning
• Stability Analysis of Deep Reinforcement Learning for Multi-Agent Inspection in a Terrestrial Testbed