Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

Abstract

We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in simulated, photo-realistic environments.

arXiv commentEMNLP 2020

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• On Evaluation of Embodied Navigation Agents
• Self-Critical Reasoning for Robust Visual Question Answering
• Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
• VALAN: Vision and Language Agent Navigation

Cited by in corpus (3)

• Visually grounded models of spoken language: A survey of datasets, architectures and evaluation techniques
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information
• Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning