Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation

Abstract

Vision-and-Language Navigation (VLN) requires grounding instructions, such as "turn right and stop at the door", to routes in a visual environment. The actual grounding can connect language to the environment through multiple modalities, e.g. "stop at the door" might ground into visual objects, while "turn right" might rely only on the geometric structure of a route. We investigate where the natural language empirically grounds under two recent state-of-the-art VLN models. Surprisingly, we discover that visual features may actually hurt these models: models which only use route structure, ablating visual features, outperform their visual counterparts in unseen new environments on the benchmark Room-to-Room dataset. To better use all the available modalities, we propose to decompose the grounding procedure into a set of expert models with access to different modalities (including object detections) and ensemble them at prediction time, improving the performance of state-of-the-art models on the VLN task.

arXiv commentACL 2019

References in corpus (9)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning to Navigate in Complex Environments
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Speaker-Follower Models for Vision-and-Language Navigation
• Women also Snowboard: Overcoming Bias in Captioning Models
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Learning to Navigate in Cities Without a Map
• Blindfold Baselines for Embodied QA
• Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments

Cited by in corpus (6)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• VTNet: Visual Transformer Network for Object Goal Navigation
• Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning
• Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
• Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation
• Bridging the visual gap in VLN via semantically richer instructions