Multi-View Learning for Vision-and-Language Navigation

Abstract

Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified. In this paper, we present a novel training paradigm, Learn from EveryOne (LEO), which leverages multiple instructions (as different views) for the same trajectory to resolve language ambiguity and improve generalization. By sharing parameters across instructions, our approach learns more effectively from limited training data and generalizes better in unseen environments. On the recent Room-to-Room (R2R) benchmark dataset, LEO achieves 16% improvement (absolute) over a greedy agent as the base agent (25.3% $\to$ 41.4%) in Success Rate weighted by Path Length (SPL). Further, LEO is complementary to most existing models for vision-and-language navigation, allowing for easy integration with the existing techniques, leading to LEO+, which creates the new state of the art, pushing the R2R benchmark to 62% (9% absolute improvement).

arXiv comment16 pages, 8 figures

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• A Survey on Multi-view Learning
• On Evaluation of Embodied Navigation Agents
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks

Cited by in corpus (5)

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Deep Learning for Embodied Vision Navigation: A Survey
• Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information