Grounding Language to Autonomously-Acquired Skills via Goal Generation

Abstract

We are interested in the autonomous acquisition of repertoires of skills. Language-conditioned reinforcement learning (LC-RL) approaches are great tools in this quest, as they allow to express abstract goals as sets of constraints on the states. However, most LC-RL agents are not autonomous and cannot learn without external instructions and feedback. Besides, their direct language condition cannot account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input. To resolve these issues, we propose a new conceptual approach to language-conditioned RL: the Language-Goal-Behavior architecture (LGB). LGB decouples skill learning and language grounding via an intermediate semantic representation of the world. To showcase the properties of LGB, we present a specific implementation called DECSTR. DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects. In a first stage (G -> B), it freely explores its environment and targets self-generated semantic configurations. In a second stage (L -> G), it trains a language-conditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs. We showcase the additional properties of LGB w.r.t. both an end-to-end LC-RL approach and a similar approach leveraging non-semantic, continuous intermediate representations. Intermediate semantic representations help satisfy language commands in a diversity of ways, enable strategy switching after a failure and facilitate language grounding.

arXiv commentPublished at ICLR 2021

References in corpus (14)

• Visual Reinforcement Learning with Imagined Goals
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Grounded Language Learning in a Simulated 3D World
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• HIGhER : Improving instruction following with Hindsight Generation for Experience Replay
• Contextual Imagined Goals for Self-Supervised Robotic Learning
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Deep Sets for Generalization in RL

Cited by in corpus (4)

• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Grounding Spatio-Temporal Language with Transformers
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Self-supervised Reinforcement Learning with Independently Controllable Subgoals