Discovery of Options via Meta-Learned Subgoals

Abstract

Temporal abstractions in the form of options have been shown to help reinforcement learning (RL) agents learn faster. However, despite prior work on this topic, the problem of discovering options through interaction with an environment remains a challenge. In this paper, we introduce a novel meta-gradient approach for discovering useful options in multi-task RL environments. Our approach is based on a manager-worker decomposition of the RL agent, in which a manager maximises rewards from the environment by learning a task-dependent policy over both a set of task-independent discovered-options and primitive actions. The option-reward and termination functions that define a subgoal for each option are parameterised as neural networks and trained via meta-gradients to maximise their usefulness. Empirical analysis on gridworld and DeepMind Lab tasks show that: (1) our approach can discover meaningful and diverse temporally-extended options in multi-task RL domains, (2) the discovered options are frequently used by the agent while learning to solve the training tasks, and (3) that the discovered options help a randomly initialised manager learn faster in completely new tasks.

References in corpus (18)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Variational Intrinsic Control
• Deep Exploration via Randomized Value Functions
• Meta Learning Shared Hierarchies
• Learning and Transfer of Modulated Locomotor Controllers
• A Laplacian Framework for Option Discovery in Reinforcement Learning
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• Adaptive Skills, Adaptive Partitions (ASAP)
• Discovery of Useful Questions as Auxiliary Tasks
• A Self-Tuning Actor-Critic Algorithm
• Compositional Planning Using Optimal Option Models
• Learning Purposeful Behaviour in the Absence of Rewards
• The Termination Critic
• What Can Learned Intrinsic Rewards Capture?

Cited by in corpus (2)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Average-Reward Learning and Planning with Options