Temporal Abstraction in Reinforcement Learning with the Successor Representation

Abstract

Reasoning at multiple levels of temporal abstraction is one of the key attributes of intelligence. In reinforcement learning, this is often modeled through temporally extended courses of actions called options. Options allow agents to make predictions and to operate at different levels of abstraction within an environment. Nevertheless, approaches based on the options framework often start with the assumption that a reasonable set of options is known beforehand. When this is not the case, there are no definitive answers for which options one should consider. In this paper, we argue that the successor representation (SR), which encodes states based on the pattern of state visitation that follows them, can be seen as a natural substrate for the discovery and use of temporal abstractions. To support our claim, we take a big picture view of recent results, showing how the SR can be used to discover options that facilitate either temporally-extended exploration or planning. We cast these results as instantiations of a general framework for option discovery in which the agent's representation is used to identify useful options, which are then used to further improve its representation. This results in a virtuous, never-ending, cycle in which both the representation and the options are constantly refined based on each other. Beyond option discovery itself, we also discuss how the SR allows us to augment a set of options into a combinatorially large counterpart without additional learning. This is achieved through the combination of previously learned options. Our empirical evaluation focuses on options discovered for exploration and on the use of the SR to combine them. The results of our experiments shed light on important design decisions involved in the definition of options and demonstrate the synergy of different methods based on the SR, such as eigenoptions and the option keyboard.

This is the final, published JMLR version

References in corpus (13)

• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Learning and Transfer of Modulated Locomotor Controllers
• Deep Successor Reinforcement Learning
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Eigenoption Discovery through the Deep Successor Representation
• Learning Purposeful Behaviour in the Absence of Rewards
• Exploration--Exploitation in MDPs with Options
• APS: Active Pretraining with Successor Features
• Discovering Options for Exploration by Minimizing Cover Time
• Finding Options that Minimize Planning Time
• When Simple Exploration is Sample Efficient: Identifying Sufficient Conditions for Random Exploration to Yield PAC RL Algorithms
• Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawing