Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction

Abstract

We propose an approach for mapping natural language instructions and raw observations to continuous control of a quadcopter drone. Our model predicts interpretable position-visitation distributions indicating where the agent should go during execution and where it should stop, and uses the predicted distributions to select the actions to execute. This two-step model decomposition allows for simple and efficient training using a combination of supervised learning and imitation learning. We evaluate our approach with a realistic drone simulator, and demonstrate absolute task-completion accuracy improvements of 16.85% over two state-of-the-art instruction-following methods.

arXiv commentAppeared in Conference on Robot Learning 2018

Cited by in corpus (16)

• Vision-and-Dialog Navigation
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• FILM: Following Instructions in Language with Modular Methods
• Safe Reinforcement Learning with Natural Language Constraints
• Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following
• Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
• Where Are You? Localization from Embodied Dialog
• Executing Instructions in Situated Collaborative Interactions
• Robust and Interpretable Grounding of Spatial References with Relation Networks
• The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation
• Spatial Language Understanding for Object Search in Partially Observed City-scale Environments
• Compositional Networks Enable Systematic Generalization for Grounded Language Understanding