Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning

Abstract

Complex, multi-task problems have proven to be difficult to solve efficiently in a sparse-reward reinforcement learning setting. In order to be sample efficient, multi-task learning requires reuse and sharing of low-level policies. To facilitate the automatic decomposition of hierarchical tasks, we propose the use of step-by-step human demonstrations in the form of natural language instructions and action trajectories. We introduce a dataset of such demonstrations in a crafting-based grid world. Our model consists of a high-level language generator and low-level policy, conditioned on language. We find that human demonstrations help solve the most complex tasks. We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations. Generalization is not only reflected in the actions of the agent, but also in the generated natural language instructions in unseen tasks. Our approach also gives our trained agent interpretable behaviors because it is able to generate a sequence of high-level descriptions of its actions.

arXiv commentAccepted at ICLR 2021

References in corpus (12)

• Modular Multitask Reinforcement Learning with Policy Sketches
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Learning and Transfer of Modulated Locomotor Controllers
• Language as an Abstraction for Hierarchical Deep Reinforcement Learning
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning
• Hierarchical Decision Making by Generating and Following Natural Language Instructions
• HIGhER : Improving instruction following with Hindsight Generation for Experience Replay
• Word2vec to behavior: morphology facilitates the grounding of language in machines
• Zero-Shot Compositional Policy Learning via Language Grounding

Cited by in corpus (2)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment