Grounded Language Learning Fast and Slow

Abstract

Recent work has shown that large text-based neural language models, trained with conventional supervised learning objectives, acquire a surprising propensity for few- and one-shot learning. Here, we show that an embodied agent situated in a simulated 3D world, and endowed with a novel dual-coding external memory, can exhibit similar one-shot word learning when trained with conventional reinforcement learning algorithms. After a single introduction to a novel object via continuous visual perception and a language prompt ("This is a dax"), the agent can re-identify the object and manipulate it as instructed ("Put the dax on the bed"). In doing so, it seamlessly integrates short-term, within-episode knowledge of the appropriate referent for the word "dax" with long-term lexical and motor knowledge acquired across episodes (i.e. "bed" and "putting"). We find that, under certain training conditions and with a particular memory writing mechanism, the agent's one-shot word-object binding generalizes to novel exemplars within the same ShapeNet category, and is effective in settings with unfamiliar numbers of objects. We further show how dual-coding memory can be exploited as a signal for intrinsic motivation, stimulating the agent to seek names for objects that may be useful for later executing instructions. Together, the results demonstrate that deep neural networks can exploit meta-learning, episodic memory and an explicitly multi-modal environment to account for 'fast-mapping', a fundamental pillar of human cognitive development and a potentially transformative capacity for agents that interact with human users.

References in corpus (17)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Language Models are Few-Shot Learners
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Agent57: Outperforming the Atari Human Benchmark
• Grounded Language Learning in a Simulated 3D World
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Stabilizing Transformers for Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Never Give Up: Learning Directed Exploration Strategies
• Meta-learning for Few-shot Natural Language Processing: A Survey
• Environmental drivers of systematicity and generalization in a situated agent
• Compositional generalization through meta sequence-to-sequence learning
• Generalization of Reinforcement Learners with Working and Episodic Memory
• Understanding Early Word Learning in Situated Artificial Agents

Cited by in corpus (9)

• Abstraction and Analogy-Making in Artificial Intelligence
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Emergent Symbols through Binding in External Memory
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning
• SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Symbolic Behaviour in Artificial Intelligence
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark