Understanding Early Word Learning in Situated Artificial Agents

Abstract

Neural network-based systems can now learn to locate the referents of words and phrases in images, answer questions about visual scenes, and execute symbolic instructions as first-person actors in partially-observable worlds. To achieve this so-called grounded language learning, models must overcome challenges that infants face when learning their first words. While it is notable that models with no meaningful prior knowledge overcome these obstacles, researchers currently lack a clear understanding of how they do so, a problem that we attempt to address in this paper. For maximum control and generality, we focus on a simple neural network-based language learning agent, trained via policy-gradient methods, which can interpret single-word instructions in a simulated 3D world. Whilst the goal is not to explicitly model infant word learning, we take inspiration from experimental paradigms in developmental psychology and apply some of these to the artificial agent, exploring the conditions under which established human biases and learning effects emerge. We further propose a novel method for visualising semantic representations in the agent.

References in corpus (11)

• Understanding Neural Networks Through Deep Visualization
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Grounded Language Learning in a Simulated 3D World
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Learning to Understand Goal Specifications by Modelling Reward
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks

Cited by in corpus (10)

• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Environmental drivers of systematicity and generalization in a situated agent
• Grounded Language Learning Fast and Slow
• Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning
• Emergence of Numeric Concepts in Multi-Agent Autonomous Communication
• A Visuospatial Dataset for Naturalistic Verb Learning
• Attention Based Natural Language Grounding by Navigating Virtual Environment
• Fast Concept Mapping: The Emergence of Human Abilities in Artificial Neural Networks when Learning Embodied and Self-Supervised
• Grounding Complex Navigational Instructions Using Scene Graphs
• The Emergence of the Shape Bias Results from Communicative Efficiency