Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation

Abstract

Deep Learning has revolutionized our ability to solve complex problems such as Vision-and-Language Navigation (VLN). This task requires the agent to navigate to a goal purely based on visual sensory inputs given natural language instructions. However, prior works formulate the problem as a navigation graph with a discrete action space. In this work, we lift the agent off the navigation graph and propose a more complex VLN setting in continuous 3D reconstructed environments. Our proposed setting, Robo-VLN, more closely mimics the challenges of real world navigation. Robo-VLN tasks have longer trajectory lengths, continuous action spaces, and challenges such as obstacles. We provide a suite of baselines inspired by state-of-the-art works in discrete VLN and show that they are less effective at this task. We further propose that decomposing the task into specialized high- and low-level policies can more effectively tackle this task. With extensive experiments, we show that by using layered decision making, modularized training, and decoupling reasoning and imitation, our proposed Hierarchical Cross-Modal (HCM) agent outperforms existing baselines in all key metrics and sets a new benchmark for Robo-VLN.

arXiv commentAccepted to ICRA 2021; Code and Dataset are available at https://github.com/GT-RIPL/robo-vln

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• On Evaluation of Embodied Navigation Agents
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Building Generalizable Agents with a Realistic and Rich 3D Environment
• PyRobot: An Open-source Robotics Framework for Research and Benchmarking
• FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning
• General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
• Exploration--Exploitation in MDPs with Options