From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Abstract

Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.

Cited by in corpus (30)

• On the Opportunities and Risks of Foundation Models
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Learning to Generalize from Sparse and Underspecified Rewards
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Reward-rational (implicit) choice: A unifying formalism for reward learning
• BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments
• General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
• Understanding Early Word Learning in Situated Artificial Agents
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• ELLA: Exploration through Learned Language Abstraction
• Learning a Prior over Intent via Meta-Inverse Reinforcement Learning
• Safe Reinforcement Learning with Natural Language Constraints
• Language Grounding through Social Interactions and Curiosity-Driven Multi-Goal Learning
• Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning
• A Narration-based Reward Shaping Approach using Grounded Natural Language Commands
• Deep Sets for Generalization in RL
• Interactive Learning from Activity Description
• An Overview of Natural Language State Representation for Reinforcement Learning
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents
• Using Logical Specifications of Objectives in Multi-Objective Reinforcement Learning
• Rapid Task-Solving in Novel Environments
• Generalization in Text-based Games via Hierarchical Reinforcement Learning
• Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
• Multimodal Attention Networks for Low-Level Vision-and-Language Navigation
• Compositional Networks Enable Systematic Generalization for Grounded Language Understanding