Multi-Modal Knowledge Graph Construction and Application: A Survey

Abstract

Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions. We finalize this survey with open research problems relevant to MMKGs.

20 pages, 8 figures, 6 tables. Accepted by TKDE 2022

References in corpus (14)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Dual-Path Convolutional Image-Text Embeddings with Instance Loss
• Knowledge Graph Embedding for Link Prediction: A Comparative Analysis
• Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• Multimodal Entity Linking for Tweets
• HAKE: Human Activity Knowledge Engine
• End-to-End Entity Classification on Multimodal Knowledge Graphs
• Relational Reasoning using Prior Knowledge for Visual Captioning
• AliMe MKG: A Multi-modal Knowledge Graph for Live-streaming E-commerce
• Do Dogs have Whiskers? A New Knowledge Base of hasPart Relations
• Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance

Cited by in corpus (8)

• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Construction of Knowledge Graphs: State and Challenges
• KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graph
• From Screens to Scenes: A Survey of Embodied AI in Healthcare
• Metarobotics for Industry and Society: Vision, Technologies, and Opportunities
• Association in Facial Phenotype, Gene, Disease: A Dataset for Explainable Rare Genetic Diseases Diagnosis
• Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
• VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities