Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning

Abstract

Learning policies for complex tasks that require multiple different skills is a major challenge in reinforcement learning (RL). It is also a requirement for its deployment in real-world scenarios. This paper proposes a novel framework for efficient multi-task reinforcement learning. Our framework trains agents to employ hierarchical policies that decide when to use a previously learned policy and when to learn a new skill. This enables agents to continually acquire new skills during different stages of training. Each learned task corresponds to a human language description. Because agents can only access previously learned skills through these descriptions, the agent can always provide a human-interpretable description of its choices. In order to help the agent learn the complex temporal dependencies necessary for the hierarchical policy, we provide it with a stochastic temporal grammar that modulates when to rely on previously learned skills and when to execute new skills. We validate our approach on Minecraft games designed to explicitly test the ability to reuse previously learned skills while simultaneously learning new skills.

14 pages, 6 figures

References in corpus (4)

• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Modular Multitask Reinforcement Learning with Policy Sketches
• Grounded Language Learning in a Simulated 3D World
• Language Understanding for Text-based Games Using Deep Reinforcement Learning

Cited by in corpus (13)

• Explainable Deep Reinforcement Learning: State of the Art and Challenges
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• Learning Abstract Options
• CraftAssist: A Framework for Dialogue-enabled Interactive Agents
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Why Build an Assistant in Minecraft?
• AndroidEnv: A Reinforcement Learning Platform for Android
• CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Forgetful Experience Replay in Hierarchical Reinforcement Learning from Demonstrations
• Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization
• Guided Policy Search for Parameterized Skills using Adverbs