Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

Abstract

Existing research studies on vision and language grounding for robot navigation focus on improving model-free deep reinforcement learning (DRL) models in synthetic environments. However, model-free DRL models do not consider the dynamics in the real-world environments, and they often fail to generalize to new scenes. In this paper, we take a radical approach to bridge the gap between synthetic studies and real-world practices---We propose a novel, planned-ahead hybrid reinforcement learning model that combines model-free and model-based reinforcement learning to solve a real-world vision-language navigation task. Our look-ahead module tightly integrates a look-ahead policy model with an environment model that predicts the next state and the reward. Experimental results suggest that our proposed method significantly outperforms the baselines and achieves the best on the real-world Room-to-Room dataset. Moreover, our scalable method is more generalizable when transferring to unseen environments.

arXiv comment21 pages, 7 figures, with supplementary material

References in corpus (7)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• VQA: Visual Question Answering
• Sequence Level Training with Recurrent Neural Networks
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• The Predictron: End-To-End Learning and Planning
• Visual Storytelling
• Value Prediction Network

Cited by in corpus (9)

• Chasing Ghosts: Instruction Following as Bayesian State Tracking
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Disentangling Propagation and Generation for Video Prediction
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• SafeRoute: Learning to Navigate Streets Safely in an Urban Environment
• Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning