Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

Abstract

Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.

arXiv commentContains main article and supplementaries

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Recurrent Neural Network Regularization
• Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots
• Hindsight Experience Replay
• Visual Reinforcement Learning with Imagined Goals
• Multi-Object Representation Learning with Iterative Variational Inference
• Go-Explore: a New Approach for Hard-Exploration Problems
• MONet: Unsupervised Scene Decomposition and Representation
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Curiosity Driven Exploration of Learned Disentangled Goal Spaces
• Learning to Understand Goal Specifications by Modelling Reward
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms
• Unicorn: Continual Learning with a Universal, Off-policy Agent
• Automated curricula through setter-solver interactions
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• HIGhER : Improving instruction following with Hindsight Generation for Experience Replay
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
• Guiding Policies with Language via Meta-Learning

Cited by in corpus (11)

• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Emergent Language: A Survey and Taxonomy
• Goals as Reward-Producing Programs
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Grounding Artificial Intelligence in the Origins of Human Behavior
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning
• Intelligent behavior depends on the ecological niche: Scaling up AI to human-like intelligence in socio-cultural environments
• Grounding Spatio-Temporal Language with Transformers
• Interactive Learning from Activity Description
• Deep Sets for Generalization in RL
• Multi-Agent Reinforcement Learning as a Computational Tool for Language Evolution Research: Historical Context and Future Challenges