Option Discovery in Hierarchical Reinforcement Learning using Spatio-Temporal Clustering

Abstract

This paper introduces an automated skill acquisition framework in reinforcement learning which involves identifying a hierarchical description of the given task in terms of abstract states and extended actions between abstract states. Identifying such structures present in the task provides ways to simplify and speed up reinforcement learning algorithms. These structures also help to generalize such algorithms over multiple tasks without relearning policies from scratch. We use ideas from dynamical systems to find metastable regions in the state space and associate them with abstract states. The spectral clustering algorithm PCCA+ is used to identify suitable abstractions aligned to the underlying structure. Skills are defined in terms of the sequence of actions that lead to transitions between such abstract states. The connectivity information from PCCA+ is used to generate these skills or options. These skills are independent of the learning task and can be efficiently reused across a variety of tasks defined over the same model. This approach works well even without the exact model of the environment by using sample trajectories to construct an approximate estimate. We also present our approach to scaling the skill acquisition framework to complex tasks with large state spaces for which we perform state aggregation using the representation learned from an action conditional video prediction network and use the skill acquisition framework on the aggregated state space.

Revised version of ICML 16 Abstraction in Reinforcement Learning workshop paper

References in corpus (3)

• Asynchronous Methods for Deep Reinforcement Learning
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• The Thing That We Tried Didn't Work Very Well : Deictic Representation in Reinforcement Learning

Cited by in corpus (9)

• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Multi-Level Discovery of Deep Options
• Visual pathways from the perspective of cost functions and multi-task deep neural networks
• Modeling Long-horizon Tasks as Sequential Interaction Landscapes
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous
• Diversity-Driven Extensible Hierarchical Reinforcement Learning
• Deep Reinforcement Learning From Raw Pixels in Doom
• Hierarchical Representation Learning for Markov Decision Processes
• Options of Interest: Temporal Abstraction with Interest Functions