Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation
arXiv:2111.05759
Abstract
Vision-and-Language Navigation (VLN) is a task that an agent is required to follow a language instruction to navigate to the goal position, which relies on the ongoing interactions with the environment during moving. Recent Transformer-based VLN methods have made great progress benefiting from the direct connections between visual observations and the language instruction via the multimodal cross-attention mechanism. However, these methods usually represent temporal context as a fixed-length vector by using an LSTM decoder or using manually designed hidden states to build a recurrent Transformer. Considering a single fixed-length vector is often insufficient to capture long-term temporal context, in this paper, we introduce Multimodal Transformer with Variable-length Memory (MTVM) for visually-grounded natural language navigation by modelling the temporal context explicitly. Specifically, MTVM enables the agent to keep track of the navigation trajectory by directly storing previous activations in a memory bank. To further boost the performance, we propose a memory-aware consistency loss to help learn a better joint representation of temporal context with random masked instructions. We evaluate MTVM on popular R2R and CVDN datasets, and our model improves Success Rate on R2R unseen validation and test set by 2% each, and reduce Goal Process by 1.6m on CVDN test set.
ECCV 2022
References in corpus (8)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
- Large-Scale Adversarial Training for Vision-and-Language Representation Learning
- Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
- History Aware Multimodal Transformer for Vision-and-Language Navigation
- Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
- On the Evaluation of Vision-and-Language Navigation Instructions