Interactive Grounded Language Acquisition and Generalization in a 2D World

Abstract

We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to receive rewards. It interactively learns the teacher's language from scratch based on two language use cases: sentence-directed navigation and question answering. It learns simultaneously the visual representations of the world, the language, and the action control. By disentangling language grounding from other computational routines and sharing a concept detection function between language grounding and prediction, the agent reliably interpolates and extrapolates to interpret sentences that contain new word combinations or new words missing from training sentences. The new words are transferred from the answers of language prediction. Such a language ability is trained and evaluated on a population of over 1.6 million distinct sentences consisting of 119 object words, 8 color words, 9 spatial-relation words, and 50 grammatical words. The proposed model significantly outperforms five comparison methods for interpreting zero-shot sentences. In addition, we demonstrate human-interpretable intermediate outputs of the model in the appendix.

arXiv commentICLR 2018 (Figure 6 caption improved)

References in corpus (10)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Modulating early visual processing by language
• Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic
• Modular Multitask Reinforcement Learning with Policy Sketches
• Towards Deep Symbolic Reinforcement Learning
• Learning to Compose Neural Networks for Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• MazeBase: A Sandbox for Learning from Games

Cited by in corpus (17)

• Environmental drivers of systematicity and generalization in a situated agent
• Visual Semantic Navigation using Scene Priors
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• Mastering emergent language: learning to guide in simulated navigation
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Object-oriented Targets for Visual Navigation using Rich Semantic Representations
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• Emergence of Numeric Concepts in Multi-Agent Autonomous Communication
• Language Conditioned Imitation Learning over Unstructured Data
• Attention Based Natural Language Grounding by Navigating Virtual Environment
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning
• Automated curriculum generation for Policy Gradients from Demonstrations
• Learning to Request Guidance in Emergent Communication
• Reinforcement Learning of Implicit and Explicit Control Flow in Instructions