Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning

Abstract

Vision and voice are two vital keys for agents' interaction and learning. In this paper, we present a novel indoor navigation model called Memory Vision-Voice Indoor Navigation (MVV-IN), which receives voice commands and analyzes multimodal information of visual observation in order to enhance robots' environment understanding. We make use of single RGB images taken by a first-view monocular camera. We also apply a self-attention mechanism to keep the agent focusing on key areas. Memory is important for the agent to avoid repeating certain tasks unnecessarily and in order for it to adapt adequately to new scenes, therefore, we make use of meta-learning. We have experimented with various functional features extracted from visual observation. Comparative experiments prove that our methods outperform state-of-the-art baselines.

arXiv comment8 pages, 6 figures, 2 tables, accepted at 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2020)

References in corpus (11)

• The Cityscapes Dataset for Semantic Urban Scene Understanding
• On Evaluation of Embodied Navigation Agents
• AI2-THOR: An Interactive 3D Environment for Visual AI
• wav2letter++: The Fastest Open-source Speech Recognition System
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Semi-parametric Topological Memory for Navigation
• Visual Semantic Navigation using Scene Priors
• Interactive Grounded Language Acquisition and Generalization in a 2D World
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Learning and Planning with a Semantic Model