Bridging Visual Perception with Contextual Semantics for Understanding Robot Manipulation Tasks

Abstract

Understanding manipulation scenarios allows intelligent robots to plan for appropriate actions to complete a manipulation task successfully. It is essential for intelligent robots to semantically interpret manipulation knowledge by describing entities, relations and attributes in a structural manner. In this paper, we propose an implementing framework to generate high-level conceptual dynamic knowledge graphs from video clips. A combination of a Vision-Language model and an ontology system, in correspondence with visual perception and contextual semantics, is used to represent robot manipulation knowledge with Entity-Relation-Entity (E-R-E) and Entity-Attribute-Value (E-A-V) tuples. The proposed method is flexible and well-versed. Using the framework, we present a case study where robot performs manipulation actions in a kitchen environment, bridging visual perception with contextual semantics using the generated dynamic knowledge graphs.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation
• Manipulation Motion Taxonomy and Coding for Robots
• Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream