Hierarchical Decision Making by Generating and Following Natural Language Instructions

Abstract

We explore using latent natural language instructions as an expressive and compositional representation of complex actions for hierarchical decision making. Rather than directly selecting micro-actions, our agent first generates a latent plan in natural language, which is then executed by a separate model. We introduce a challenging real-time strategy game environment in which the actions of a large number of units must be coordinated across long time scales. We gather a dataset of 76 thousand pairs of instructions and executions from human play, and train instructor and executor models. Experiments show that models using natural language as a latent variable significantly outperform models that directly imitate human actions. The compositional structure of language proves crucial to its effectiveness for action representation. We also release our code, models and data.

References in corpus (15)

• Playing Atari with Deep Reinforcement Learning
• StarCraft II: A New Challenge for Reinforcement Learning
• Modular Multitask Reinforcement Learning with Policy Sketches
• Relational Deep Reinforcement Learning
• Grounded Language Learning in a Simulated 3D World
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Episodic Exploration for Deep Deterministic Policies: An Application to StarCraft Micromanagement Tasks
• TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• Hierarchical Text Generation and Planning for Strategic Dialogue
• MazeBase: A Sandbox for Learning from Games
• Implementing a Wall-In Building Placement in StarCraft with Declarative Programming
• Unified Pragmatic Models for Generating and Following Instructions

Cited by in corpus (9)

• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• I love your chain mail! Making knights smile in a fantasy game world: Open-domain goal-oriented dialogue agents
• Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data
• Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
• Computational Rational Engineering and Development: Synergies and Opportunities
• NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment
• Executing Instructions in Situated Collaborative Interactions
• Generalizable and Explainable Dialogue Generation via Explicit Action Learning
• Guided Policy Search for Parameterized Skills using Adverbs