Learning Robust Options

Abstract

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally abstract actions, in the framework of options. We present a Robust Options Policy Iteration (ROPI) algorithm with convergence guarantees, which learns options that are robust to model uncertainty. We utilize ROPI to learn robust options with the Robust Options Deep Q Network (RO-DQN) that solves multiple tasks and mitigates model misspecification due to model uncertainty. We present experimental results which suggest that policy iteration with linear features may have an inherent form of robustness when using coarse feature representations. In addition, we present experimental results which demonstrate that robustness helps policy iteration implemented on top of deep neural networks to generalize over a much broader range of dynamics than non-robust policy iteration.

Cited by in corpus (12)

• Challenges of Real-World Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Robust Reinforcement Learning on State Observations with Learned Optimal Adversary
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• A Bayesian Approach to Robust Reinforcement Learning
• TempoRL: Learning When to Act
• DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics
• Twice regularized MDPs and the equivalence between robustness and regularization
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Learning Robust Options by Conditional Value at Risk Optimization
• Learning Robust Controllers Via Probabilistic Model-Based Policy Search