Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation

Abstract

The ability to perform effective planning is crucial for building an instruction-following agent. When navigating through a new environment, an agent is challenged with (1) connecting the natural language instructions with its progressively growing knowledge of the world; and (2) performing long-range planning and decision making in the form of effective exploration and error correction. Current methods are still limited on both fronts despite extensive efforts. In this paper, we introduce the Evolving Graphical Planner (EGP), a model that performs global planning for navigation based on raw sensory input. The model dynamically constructs a graphical representation, generalizes the action space to allow for more flexible decision making, and performs efficient planning on a proxy graph representation. We evaluate our model on a challenging Vision-and-Language Navigation (VLN) task with photorealistic images and achieve superior performance compared to previous navigation architectures. For instance, we achieve a 53% success rate on the test split of the Room-to-Room navigation task through pure imitation learning, outperforming previous navigation architectures by up to 5%.

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Deep Convolutional Networks on Graph-Structured Data
• Gated Graph Sequence Neural Networks
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Professor Forcing: A New Algorithm for Training Recurrent Networks
• Modular Multitask Reinforcement Learning with Policy Sketches
• Vision-and-Dialog Navigation
• Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning
• Neural Map: Structured Memory for Deep Reinforcement Learning
• From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
• Semi-parametric Topological Memory for Navigation
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
• Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-training
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• RMM: A Recursive Mental Model for Dialog Navigation
• Hallucinative Topological Memory for Zero-Shot Visual Planning
• Robust Navigation with Language Pretraining and Stochastic Sampling
• Multi-View Learning for Vision-and-Language Navigation
• Learning to cooperate: Emergent communication in multi-agent navigation
• Optimal Completion Distillation for Sequence Learning

Cited by in corpus (6)

• Collaborative Visual Navigation
• Topological Planning with Transformers for Vision-and-Language Navigation
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory
• SGoLAM: Simultaneous Goal Localization and Mapping for Multi-Object Goal Navigation
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments