Vision-and-Dialog Navigation

Abstract

Robots navigating in human environments should use language to ask for assistance and be able to understand human responses. To study this challenge, we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k embodied, human-human dialogs situated in simulated, photorealistic home environments. The Navigator asks questions to their partner, the Oracle, who has privileged access to the best next steps the Navigator should take according to a shortest path planner. To train agents that search an environment for a goal location, we define the Navigation from Dialog History task. An agent, given a target object and a dialog history between humans cooperating to find that object, must infer navigation actions towards the goal in unexplored environments. We establish an initial, multi-modal sequence-to-sequence model and demonstrate that looking farther back in the dialog history improves performance. Sourcecode and a live interface demo can be found at https://cvdn.dev/

arXiv commentConference on Robot Learning (CoRL) 2019

References in corpus (5)

• VQA: Visual Question Answering
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction

Cited by in corpus (13)

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Rule-Based Reinforcement Learning for Efficient Robot Navigation with Space Reduction
• DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy
• Language and Visual Entity Relationship Graph for Agent Navigation
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• Multimodal Speech Recognition for Language-Guided Embodied Agents
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning