Speaker-Follower Models for Vision-and-Language Navigation

Abstract

Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete low-level motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach---speaker-driven data augmentation, pragmatic reasoning and panoramic action space---dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

arXiv commentNIPS 2018

Cited by in corpus (26)

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigation
• A Dual Semantic-Aware Recurrent Global-Adaptive Network For Vision-and-Language Navigation
• Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
• Robust Navigation with Language Pretraining and Stochastic Sampling
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory
• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering
• MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation
• Sub-Instruction Aware Vision-and-Language Navigation
• A practical introduction to the Rational Speech Act modeling framework
• Safe Reinforcement Learning with Natural Language Constraints
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information
• Controllable Navigation Instruction Generation with Chain of Thought Prompting
• Local Slot Attention for Vision-and-Language Navigation
• Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
• Bridging the visual gap in VLN via semantically richer instructions