Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation

Abstract

We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. In order to accomplish this, humans need easy and effective ways of specifying tasks to the robot. Goal images are one popular form of task specification, as they are already grounded in the robot's observation space. However, goal images also have a number of drawbacks: they are inconvenient for humans to provide, they can over-specify the desired behavior leading to a sparse reward signal, or under-specify task information in the case of non-goal reaching tasks. Natural language provides a convenient and flexible alternative for task specification, but comes with the challenge of grounding language in the robot's observation space. To scalably learn this grounding we propose to leverage offline robot datasets (including highly sub-optimal, autonomously collected data) with crowd-sourced natural language labels. With this data, we learn a simple classifier which predicts if a change in state completes a language instruction. This provides a language-conditioned reward function that can then be used for offline multi-task RL. In our experiments, we find that on language-conditioned manipulation tasks our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%, and is able to perform visuomotor tasks from natural language, such as "open the right drawer" and "move the stapler", on a Franka Emika Panda robot.

Conference on Robot Learning (CoRL) 2021. 24 Pages, 18 Figures

References in corpus (29)

• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Hindsight Experience Replay
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control
• Modular Multitask Reinforcement Learning with Policy Sketches
• Grounded Language Learning in a Simulated 3D World
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Stochastic Variational Video Prediction
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
• Learning to Understand Goal Specifications by Modelling Reward
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• Beating Atari with Natural Language Guided Reinforcement Learning
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• HIGhER : Improving instruction following with Hindsight Generation for Experience Replay
• Experience-Embedded Visual Foresight
• PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards
• Topological Planning with Transformers for Vision-and-Language Navigation
• ELLA: Exploration through Learned Language Abstraction
• Guiding Policies with Language via Meta-Learning
• Following Instructions by Imagining and Reaching Visual Goals