A Recurrent Vision-and-Language BERT for Navigation

Abstract

Accuracy of many visiolinguistic tasks has benefited significantly from the application of vision-and-language(V&L) BERT. However, its application for the task of vision-and-language navigation (VLN) remains limited. One reason for this is the difficulty adapting the BERT architecture to the partially observable Markov decision process present in VLN, requiring history-dependent attention and decision making. In this paper we propose a recurrent BERT model that is time-aware for use in VLN. Specifically, we equip the BERT model with a recurrent function that maintains cross-modal state information for the agent. Through extensive experiments on R2R and REVERIE we demonstrate that our model can replace more complex encoder-decoder models to achieve state-of-the-art results. Moreover, our approach can be generalised to other transformer-based architectures, supports pre-training, and is capable of solving navigation and referring expression tasks simultaneously.

References in corpus (6)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Decoupled Weight Decay Regularization
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• On Evaluation of Embodied Navigation Agents
• CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
• OmniNet: A unified architecture for multi-modal multi-task learning

Cited by in corpus (7)

• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• Language Grounding with 3D Objects
• Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
• Neighbor-view Enhanced Model for Vision and Language Navigation
• Local Slot Attention for Vision-and-Language Navigation