Chasing Ghosts: Instruction Following as Bayesian State Tracking

Abstract

A visually-grounded navigation instruction can be interpreted as a sequence of expected observations and actions an agent following the correct trajectory would encounter and perform. Based on this intuition, we formulate the problem of finding the goal location in Vision-and-Language Navigation (VLN) within the framework of Bayesian state tracking - learning observation and motion models conditioned on these expectable events. Together with a mapper that constructs a semantic spatial map on-the-fly during navigation, we formulate an end-to-end differentiable Bayes filter and train it to identify the goal by predicting the most likely trajectory through the map according to the instructions. The resulting navigation policy constitutes a new approach to instruction following that explicitly models a probability distribution over states, encoding strong geometric and algorithmic priors while enabling greater explainability. Our experiments show that our approach outperforms a strong LingUNet baseline when predicting the goal location on the map. On the full VLN task, i.e. navigating to the goal location, our approach achieves promising results with less reliance on navigation constraints.

arXiv commentNeurIPS 2019

References in corpus (12)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• On Evaluation of Embodied Navigation Agents
• Learning to Navigate in Complex Environments
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Control of Memory, Active Perception, and Action in Minecraft
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
• Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Prediction
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation

Cited by in corpus (9)

• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• Language-Conditioned Imitation Learning for Robot Manipulation Tasks
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• SGoLAM: Simultaneous Goal Localization and Mapping for Multi-Object Goal Navigation
• Verbal Focus-of-Attention System for Learning-from-Observation
• Where Are You? Localization from Embodied Dialog