Learning Abstract Options

Abstract

Building systems that autonomously create temporal abstractions from data is a key challenge in scaling learning and planning in reinforcement learning. One popular approach for addressing this challenge is the options framework (Sutton et al., 1999). However, only recently in (Bacon et al., 2017) was a policy gradient theorem derived for online learning of general purpose options in an end to end fashion. In this work, we extend previous work on this topic that only focuses on learning a two-level hierarchy including options and primitive actions to enable learning simultaneously at multiple resolutions in time. We achieve this by considering an arbitrarily deep hierarchy of options where high level temporally extended options are composed of lower level options with finer resolutions in time. We extend results from (Bacon et al., 2017) and derive policy gradient theorems for a deep hierarchy of options. Our proposed hierarchical option-critic architecture is capable of learning internal policies, termination conditions, and hierarchical compositions over options without the need for any intrinsic rewards or subgoals. Our empirical results in both discrete and continuous environments demonstrate the efficiency of our framework.

arXiv commentNeurIPS 2018

References in corpus (11)

• Overcoming catastrophic forgetting in neural networks
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Strategic Attentive Writer for Learning Macro-Actions
• Multi-Level Discovery of Deep Options
• Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
• Adaptive Skills, Adaptive Partitions (ASAP)
• Learnings Options End-to-End for Continuous Action Tasks
• Learning to Compose Skills
• Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
• Learning to Multi-Task by Active Sampling

Cited by in corpus (9)

• Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
• A survey on intrinsic motivation in reinforcement learning
• Hierarchical Reinforcement Learning By Discovering Intrinsic Options
• Safe Option-Critic: Learning Safety in the Option-Critic Architecture
• CompILE: Compositional Imitation Learning and Execution
• Learning Hierarchical Teaching Policies for Cooperative Agents
• Data-efficient Hindsight Off-policy Option Learning
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks
• Coagent Networks Revisited