Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

Abstract

Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• Grounded Language Learning in a Simulated 3D World
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• No Training Required: Exploring Random Encoders for Sentence Classification
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Combating Adversarial Misspellings with Robust Word Recognition
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• RTFM: Generalising to Novel Environment Dynamics via Reading

Cited by in corpus (14)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Aligning AI With Shared Human Values
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• Language Grounding with 3D Objects
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• WordCraft: An Environment for Benchmarking Commonsense Agents
• AlignNet: Unsupervised Entity Alignment
• ELLA: Exploration through Learned Language Abstraction
• Probing Emergent Semantics in Predictive Agents via Question Answering
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Learning Controllable Content Generators
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Reinforcement Learning of Implicit and Explicit Control Flow in Instructions