Safe Option-Critic: Learning Safety in the Option-Critic Architecture

Abstract

Designing hierarchical reinforcement learning algorithms that exhibit safe behaviour is not only vital for practical applications but also, facilitates a better understanding of an agent's decisions. We tackle this problem in the options framework, a particular way to specify temporally abstract actions which allow an agent to use sub-policies with start and end conditions. We consider a behaviour as safe that avoids regions of state-space with high uncertainty in the outcomes of actions. We propose an optimization objective that learns safe options by encouraging the agent to visit states with higher behavioural consistency. The proposed objective results in a trade-off between maximizing the standard expected return and minimizing the effect of model uncertainty in the return. We propose a policy gradient algorithm to optimize the constrained objective function. We examine the quantitative and qualitative behaviour of the proposed approach in a tabular grid-world, continuous-state puddle-world, and three games from the Arcade Learning Environment: Ms.Pacman, Amidar, and Q*Bert. Our approach achieves a reduction in the variance of return, boosts performance in environments with intrinsic variability in the reward structure, and compares favorably both with primitive actions as well as with risk-neutral options.

arXiv commentTo appear at The Knowledge Engineering Review (KER), 2021. Previous draft appeared in Adaptive Learning Agents (ALA) 2018 workshop held at ICML, AAMAS in Stockholm. Corrected typos, added references and added extra figures

References in corpus (13)

• Dueling Network Architectures for Deep Reinforcement Learning
• Asynchronous Methods for Deep Reinforcement Learning
• The Arcade Learning Environment: An Evaluation Platform for General Agents
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• On the Complexity of Best Arm Identification in Multi-Armed Bandit Models
• Massively Parallel Methods for Deep Reinforcement Learning
• Risk-Sensitive Reinforcement Learning Applied to Control under Constraints
• Policy Gradients with Variance Related Risk Criteria
• Strategic Attentive Writer for Learning Macro-Actions
• Inverse Reward Design
• Learning Abstract Options
• Adaptive Skills, Adaptive Partitions (ASAP)
• Scaling Up Robust MDPs by Reinforcement Learning

Cited by in corpus (4)

• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Reimagining an autonomous vehicle
• Reinforcement Learning Under Algorithmic Triage
• Transfer Learning by Modeling a Distribution over Policies