Learning to Understand Goal Specifications by Modelling Reward

Abstract

Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations---and for instructions---not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.

arXiv comment19 pages, 9 figures

Cited by in corpus (32)

• On the Opportunities and Risks of Foundation Models
• Scalable agent alignment via reward modeling: a research direction
• Learning to Generalize from Sparse and Underspecified Rewards
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals
• Understanding Early Word Learning in Situated Artificial Agents
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• LTL2Action: Generalizing LTL Instructions for Multi-Task RL
• Open Problems in Cooperative AI
• PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards
• Adversarially Guided Actor-Critic
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Quantifying Differences in Reward Functions
• ELLA: Exploration through Learned Language Abstraction
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• Mastering emergent language: learning to guide in simulated navigation
• What Can Learned Intrinsic Rewards Capture?
• Following Instructions by Imagining and Reaching Visual Goals
• Deep Sets for Generalization in RL
• Grounding Spatio-Temporal Language with Transformers
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Connecting Context-specific Adaptation in Humans to Meta-learning
• An Extensible Interactive Interface for Agent Design
• Exploiting Language Instructions for Interpretable and Compositional Reinforcement Learning
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Hindsight Reward Tweaking via Conditional Deep Reinforcement Learning
• Learning What To Do by Simulating the Past