VSGM -- Enhance robot task understanding ability through visual semantic graph

Abstract

In recent years, developing AI for robotics has raised much attention. The interaction of vision and language of robots is particularly difficult. We consider that giving robots an understanding of visual semantics and language semantics will improve inference ability. In this paper, we propose a novel method-VSGM (Visual Semantic Graph Memory), which uses the semantic graph to obtain better visual image features, improve the robot's visual understanding ability. By providing prior knowledge of the robot and detecting the objects in the image, it predicts the correlation between the attributes of the object and the objects and converts them into a graph-based representation; and mapping the object in the image to be a top-down egocentric map. Finally, the important object features of the current task are extracted by Graph Neural Networks. The method proposed in this paper is verified in the ALFRED (Action Learning From Realistic Environments and Directives) dataset. In this dataset, the robot needs to perform daily indoor household tasks following the required language instructions. After the model is added to the VSGM, the task success rate can be improved by 6~10%.

arXiv comment16 pages, 7 figures

References in corpus (13)

• Semi-Supervised Classification with Graph Convolutional Networks
• Graph Transformer Networks
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents
• MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation
• Visual Semantic Navigation using Scene Priors
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction
• Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views
• Topological Planning with Transformers for Vision-and-Language Navigation
• Regression Planning Networks
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation