Guiding Policies with Language via Meta-Learning

Abstract

Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines. However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task. In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill. Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly. In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.

arXiv commentAccepted at ICLR 2019

References in corpus (9)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Deep reinforcement learning from human preferences
• Learning to reinforcement learn
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Learning Robust Rewards with Adversarial Inverse Reinforcement Learning
• Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
• Few-Shot Goal Inference for Visuomotor Learning and Planning
• Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Cited by in corpus (4)

• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• An Extensible Interactive Interface for Agent Design