Visual Semantic Navigation using Scene Priors

Abstract

How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects. The supplementary video can be accessed at the following link: https://youtu.be/otKjuO805dE .

References in corpus (4)

• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• segDeepM: Exploiting Segmentation and Context in Deep Neural Networks for Object Detection
• Interactive Grounded Language Acquisition and Generalization in a 2D World

Cited by in corpus (21)

• Object Goal Navigation using Goal-Oriented Semantic Exploration
• Learning hierarchical relationships for object-goal navigation
• Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers
• A Survey of Embodied AI: From Simulators to Research Tasks
• Visual Semantic Reasoning for Image-Text Matching
• A Behavioral Approach to Visual Navigation with Graph Localization Networks
• Learning Object Relation Graph and Tentative Policy for Visual Navigation
• SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
• Learning to Map for Active Semantic Goal Navigation
• Object-oriented Targets for Visual Navigation using Rich Semantic Representations
• Context-Aware Zero-Shot Recognition
• Reinforcement Learning-based Visual Navigation with Information-Theoretic Regularization
• Generalization to Novel Objects using Prior Relational Knowledge
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• NeoNav: Improving the Generalization of Visual Navigation via Generating Next Expected Observations
• Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning
• Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledge
• Efficient Robotic Object Search via HIEM: Hierarchical Policy Learning with Intrinsic-Extrinsic Modeling
• HiCoRe: Visual Hierarchical Context-Reasoning
• Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph
• VSGM -- Enhance robot task understanding ability through visual semantic graph