Language-Conditioned Imitation Learning for Robot Manipulation Tasks

Abstract

Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., "go to the large green bowl"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods.

arXiv commentAccepted to the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada as spotlight presentation

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• PyRep: Bringing V-REP to Deep Robot Learning
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Goal-conditioned Imitation Learning
• The EMPATHIC Framework for Task Learning from Implicit Human Feedback
• Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control

Cited by in corpus (5)

• Text2Motion: From Natural Language Instructions to Feasible Plans
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• Zero-shot Task Adaptation using Natural Language
• PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulation
• Language Conditioned Imitation Learning over Unstructured Data