Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning

Abstract

As a step towards developing zero-shot task generalization capabilities in reinforcement learning (RL), we introduce a new RL problem where the agent should learn to execute sequences of instructions after learning useful skills that solve subtasks. In this problem, we consider two types of generalizations: to previously unseen instructions and to longer sequences of instructions. For generalization over unseen instructions, we propose a new objective which encourages learning correspondences between similar subtasks by making analogies. For generalization over sequential instructions, we present a hierarchical architecture where a meta controller learns to use the acquired skills for executing the instructions. To deal with delayed reward, we propose a new neural architecture in the meta controller that learns when to update the subtask, which makes learning more efficient. Experimental results on a stochastic 3D domain show that the proposed ideas are crucial for generalization to longer instructions as well as unseen instructions.

ICML 2017

References in corpus (11)

• Exploiting Similarities among Languages for Machine Translation
• Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation
• Hierarchical Multiscale Recurrent Neural Networks
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Control of Memory, Active Perception, and Action in Minecraft
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Neural Turing Machines
• Learning and Transfer of Modulated Locomotor Controllers
• Policy Distillation
• Learning Parameterized Skills
• MazeBase: A Sandbox for Learning from Games

Cited by in corpus (47)

• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• Intelligent problem-solving as integrated hierarchical reinforcement learning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Composable Planning with Attributes
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning
• Understanding Early Word Learning in Situated Artificial Agents
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Zero-shot Domain Adaptation without Domain Semantic Descriptors
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• Transforming task representations to perform novel tasks
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Task-agnostic Exploration in Reinforcement Learning
• CompILE: Compositional Imitation Learning and Execution
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• Learning to Synthesize Programs as Interpretable and Generalizable Policies
• Disentangled Skill Embeddings for Reinforcement Learning
• BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
• I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Learning Invariances for Policy Generalization
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• An Overview of Natural Language State Representation for Reinforcement Learning
• Exploration via Hindsight Goal Generation
• A Conceptual Framework for Lifelong Learning
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Object-oriented state editing for HRL
• Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
• Zero-shot task adaptation by homoiconic meta-mapping
• Recursive Sketches for Modular Deep Learning
• Fast Task-Adaptation for Tasks Labeled Using Natural Language in Reinforcement Learning
• Multitasking Inhibits Semantic Drift
• Reinforcement Learning of Implicit and Explicit Control Flow in Instructions
• Task-Oriented Language Grounding for Language Input with Multiple Sub-Goals of Non-Linear Order
• Developing cooperative policies for multi-stage tasks
• gComm: An environment for investigating generalization in Grounded Language Acquisition
• Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• Guided Policy Search for Parameterized Skills using Adverbs
• Generalization in Text-based Games via Hierarchical Reinforcement Learning