Gated-Attention Architectures for Task-Oriented Language Grounding

Abstract

To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.

To appear in AAAI-18

Cited by in corpus (55)

• Attention in Natural Language Processing
• Learning Factorized Multimodal Representations
• Object Goal Navigation using Goal-Oriented Semantic Exploration
• ViZDoom Competitions: Playing Doom from Pixels
• TextWorld: A Learning Environment for Text-based Games
• Learning to Generalize from Sparse and Underspecified Rewards
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Environmental drivers of systematicity and generalization in a situated agent
• Learning to Navigate in Cities Without a Map
• Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
• FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning
• Grounded Language Learning Fast and Slow
• DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Understanding Early Word Learning in Situated Artificial Agents
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• Sim-to-Real Transfer for Vision-and-Language Navigation
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Multi-Task Reinforcement Learning with Context-based Representations
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Multimodal Grounding for Language Processing
• Embodied Question Answering in Photorealistic Environments with Point Cloud Perception
• What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• Multi-Target Embodied Question Answering
• Bayesian Relational Memory for Semantic Visual Navigation
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• NLPGym -- A toolkit for evaluating RL agents on Natural Language Processing Tasks
• Unsupervised Domain Adaptation for Visual Navigation
• SAILenv: Learning in Virtual Visual Environments Made Simple
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Interactive Learning from Activity Description
• An Overview of Natural Language State Representation for Reinforcement Learning
• Cross-View Policy Learning for Street Navigation
• Zero-Shot Compositional Policy Learning via Language Grounding
• Continual and Multi-task Reinforcement Learning With Shared Episodic Memory
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• Robot Sound Interpretation: Combining Sight and Sound in Learning-Based Control
• Grounding Complex Navigational Instructions Using Scene Graphs
• Building Intelligent Autonomous Navigation Agents
• Resolving Referring Expressions in Images With Labeled Elements
• Geometric Generalization Based Zero-Shot Learning Dataset Infinite World: Simple Yet Powerful
• Learning to Follow Language Instructions with Compositional Policies
• Attribute-Based Robotic Grasping with One-Grasp Adaptation
• Empirically Verifying Hypotheses Using Reinforcement Learning
• A new dataset and model for learning to understand navigational instructions
• Toward Human-Level Artificial Intelligence
• Dynamic Attention Networks for Task Oriented Grounding
• Task-Oriented Language Grounding for Language Input with Multiple Sub-Goals of Non-Linear Order
• TexRel: a Green Family of Datasets for Emergent Communications on Relations
• Reinforcement Learning of Implicit and Explicit Control Flow in Instructions
• Language coverage and generalization in RNN-based continuous sentence embeddings for interacting agents