Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Abstract

The Vision-and-Language Navigation (VLN) task entails an agent following navigational instruction in photo-realistic unknown environments. This challenging task demands that the agent be aware of which instruction was completed, which instruction is needed next, which way to go, and its navigation progress towards the goal. In this paper, we introduce a self-monitoring agent with two complementary components: (1) visual-textual co-grounding module to locate the instruction completed in the past, the instruction required for the next action, and the next moving direction from surrounding images and (2) progress monitor to ensure the grounded instruction correctly reflects the navigation progress. We test our self-monitoring agent on a standard benchmark and analyze our proposed approach through a series of ablation studies that elucidate the contributions of the primary components. Using our proposed method, we set the new state of the art by a significant margin (8% absolute increase in success rate on the unseen test set). Code is available at https://github.com/chihyaoma/selfmonitoring-agent .

arXiv commentICLR 2019, code is available at https://github.com/chihyaoma/selfmonitoring-agent

References in corpus (6)

• On Evaluation of Embodied Navigation Agents
• Learning to Navigate in Complex Environments
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents

Cited by in corpus (46)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• Talk2Nav: Long-Range Vision-and-Language Navigation with Dual Attention and Spatial Memory
• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding
• Language and Visual Entity Relationship Graph for Agent Navigation
• Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
• Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling
• PASTS: Progress-Aware Spatio-Temporal Transformer Speaker For Vision-and-Language Navigation
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
• Robust Navigation with Language Pretraining and Stochastic Sampling
• Multi-View Learning for Vision-and-Language Navigation
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory
• Conditional Driving from Natural Language Instructions
• PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigation
• Sub-Instruction Aware Vision-and-Language Navigation
• MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation
• Just Ask:An Interactive Learning Framework for Vision and Language Navigation
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Transferable Representation Learning in Vision-and-Language Navigation
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method
• Multimodal Text Style Transfer for Outdoor Vision-and-Language Navigation
• Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation
• Local Slot Attention for Vision-and-Language Navigation
• Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule
• Cross-Lingual Vision-Language Navigation
• Multimodal Transformer with Variable-length Memory for Vision-and-Language Navigation
• Curriculum Learning for Vision-and-Language Navigation
• Bridging the visual gap in VLN via semantically richer instructions
• Are you doing what I say? On modalities alignment in ALFRED
• Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout
• Towards Navigation by Reasoning over Spatial Configurations
• Multimodal Attention Networks for Low-Level Vision-and-Language Navigation