FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning

Abstract

Understanding and following directions provided by humans can enable robots to navigate effectively in unknown situations. We present FollowNet, an end-to-end differentiable neural architecture for learning multi-modal navigation policies. FollowNet maps natural language instructions as well as visual and depth inputs to locomotion primitives. FollowNet processes instructions using an attention mechanism conditioned on its visual and depth input to focus on the relevant parts of the command while performing the navigation task. Deep reinforcement learning (RL) a sparse reward learns simultaneously the state representation, the attention function, and control policies. We evaluate our agent on a dataset of complex natural language directions that guide the agent through a rich and realistic dataset of simulated homes. We show that the FollowNet agent learns to execute previously unseen instructions described with a similar vocabulary, and successfully navigates along paths not encountered during training. The agent shows 30% improvement over a baseline model without the attention mechanism, with 52% success rate at novel instructions.

7 pages, 8 figures

References in corpus (2)

• Grounded Language Learning in a Simulated 3D World
• Gated-Attention Architectures for Task-Oriented Language Grounding

Cited by in corpus (15)

• Spatial Action Maps for Mobile Manipulation
• Search on the Replay Buffer: Bridging Planning and Reinforcement Learning
• Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight
• Evolving Rewards to Automate Reinforcement Learning
• Conditional Driving from Natural Language Instructions
• Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following
• Enabling Robots to Understand Incomplete Natural Language Instructions Using Commonsense Reasoning
• A Narration-based Reward Shaping Approach using Grounded Natural Language Commands
• SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
• ArraMon: A Joint Navigation-Assembly Instruction Interpretation Task in Dynamic Environments
• Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
• AB-Mapper: Attention and BicNet Based Multi-agent Path Finding for Dynamic Crowded Environment
• Resolving Referring Expressions in Images With Labeled Elements
• Visual Perspective Taking for Opponent Behavior Modeling
• Compositional Networks Enable Systematic Generalization for Grounded Language Understanding