Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
arXiv:1809.00786
Abstract
We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.
arXiv commentAccepted at EMNLP 2018
References in corpus (7)
- Playing Atari with Deep Reinforcement Learning
- Instance Normalization: The Missing Ingredient for Fast Stylization
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Speaker-Follower Models for Vision-and-Language Navigation
- Grounded Language Learning in a Simulated 3D World
- Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
- CHALET: Cornell House Agent Learning Environment
Cited by in corpus (9)
- Chasing Ghosts: Instruction Following as Bayesian State Tracking
- Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
- Understanding Early Word Learning in Situated Artificial Agents
- A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
- Question-Conditioned Counterfactual Image Generation for VQA
- Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring
- Are you doing what I say? On modalities alignment in ALFRED
- VSGM -- Enhance robot task understanding ability through visual semantic graph
- Reinforcement Learning of Implicit and Explicit Control Flow in Instructions