Situational Fusion of Visual Representation for Visual Navigation

Abstract

A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the agent might need to identify a chair in a living room using semantics, follow along a hallway using vanishing point cues, and avoid obstacles using depth. Therefore, utilizing the appropriate visual perception abilities based on a situational understanding of the visual environment can empower these navigation models in unseen visual environments. We propose to train an agent to fuse a large set of visual representations that correspond to diverse visual perception abilities. To fully utilize each representation, we develop an action-level representation fusion scheme, which predicts an action candidate from each representation and adaptively consolidate these action candidates into the final action. Furthermore, we employ a data-driven inter-task affinity regularization to reduce redundancies and improve generalization. Our approach leads to a significantly improved performance in novel environments over ImageNet-pretrained baseline and other fusion methods.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• What makes ImageNet good for transfer learning?
• Driving Policy Transfer via Modularity and Abstraction
• Grounded Language Learning in a Simulated 3D World
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Neural Map: Structured Memory for Deep Reinforcement Learning
• Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight

Cited by in corpus (10)

• Cross-view Semantic Segmentation for Sensing Surroundings
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• VTNet: Visual Transformer Network for Object Goal Navigation
• Learning Object Relation Graph and Tentative Policy for Visual Navigation
• From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN)
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Environment Predictive Coding for Embodied Agents
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Learning to See before Learning to Act: Visual Pre-training for Manipulation
• Polyline Generative Navigable Space Segmentation for Autonomous Visual Navigation