Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight

Abstract

We propose a joint simulation and real-world learning framework for mapping navigation instructions and raw first-person observations to continuous control. Our model estimates the need for environment exploration, predicts the likelihood of visiting environment positions during execution, and controls the agent to both explore and visit high-likelihood positions. We introduce Supervised Reinforcement Asynchronous Learning (SuReAL). Learning uses both simulation and real environments without requiring autonomous flight in the physical environment during training, and combines supervised learning for predicting positions to visit and reinforcement learning for continuous control. We evaluate our approach on a natural language instruction-following task with a physical quadcopter, and demonstrate effective execution and exploration behavior.

Conference on Robot Learning (CoRL) 2019

References in corpus (8)

• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Grounded Language Learning in a Simulated 3D World
• Sim-to-Real Robot Learning from Pixels with Progressive Nets
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Universal Planning Networks
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction
• FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning

Cited by in corpus (9)

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
• Where Are You? Localization from Embodied Dialog
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• Spatial Language Understanding for Object Search in Partially Observed City-scale Environments
• The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation
• Learning to Follow Language Instructions with Compositional Policies