Topological Planning with Transformers for Vision-and-Language Navigation

Abstract

Conventional approaches to vision-and-language navigation (VLN) are trained end-to-end but struggle to perform well in freely traversable environments. Inspired by the robotics community, we propose a modular approach to VLN using topological maps. Given a natural language instruction and topological map, our approach leverages attention mechanisms to predict a navigation plan in the map. The plan is then executed with low-level actions (e.g. forward, rotate) using a robust controller. Experiments show that our method outperforms previous end-to-end approaches, generates interpretable navigation plans, and exhibits intelligent behaviors such as backtracking.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Language Models are Few-Shot Learners
• Learning to Navigate in Complex Environments
• Learning to Explore using Active Neural SLAM
• Learning to Act by Predicting the Future
• Unifying Map and Landmark Based Representations for Visual Navigation
• Learning Exploration Policies for Navigation
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Learning Object-conditioned Exploration using Distributed Soft Actor Critic