Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks

Abstract

Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches exploit the vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have neglected the rich semantic information contained in the environment (such as implicit navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, estimating the navigation progress, predicting the next orientation, and evaluating the trajectory consistency. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activity and build a thorough perception of the environment. Our experiments indicate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. Empirically, we demonstrate that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.

References in corpus (19)

• Distributed Representations of Words and Phrases and their Compositionality
• End to End Learning for Self-Driving Cars
• Conditional Image Synthesis With Auxiliary Classifier GANs
• VQA: Visual Question Answering
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• On Evaluation of Embodied Navigation Agents
• Learning to Navigate in Complex Environments
• Generative Adversarial Imitation Learning
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Unsupervised Domain Adaptation through Self-Supervision
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Vision-and-Dialog Navigation
• Self-Supervised Generalisation with Meta Auxiliary Learning
• Test-Time Training with Self-Supervision for Generalization under Distribution Shifts
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Robust Navigation with Language Pretraining and Stochastic Sampling

Cited by in corpus (7)

• iPlanner: Imperative Path Planning
• Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text
• From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Multi-View Learning for Vision-and-Language Navigation
• Sub-Instruction Aware Vision-and-Language Navigation
• Active Visual Information Gathering for Vision-Language Navigation