Graph Neural Networks in Vision-Language Image Understanding: A Survey

Abstract

2D image understanding is a complex problem within computer vision, but it holds the key to providing human-level scene comprehension. It goes further than identifying the objects in an image, and instead, it attempts to understand the scene. Solutions to this problem form the underpinning of a range of tasks, including image captioning, visual question answering (VQA), and image retrieval. Graphs provide a natural way to represent the relational arrangement between objects in an image, and thus, in recent years graph neural networks (GNNs) have become a standard component of many 2D image understanding pipelines, becoming a core architectural component, especially in the VQA group of tasks. In this survey, we review this rapidly evolving field and we provide a taxonomy of graph types used in 2D image understanding approaches, a comprehensive list of the GNN models used in this domain, and a roadmap of future potential developments. To the best of our knowledge, this is the first comprehensive survey that covers image captioning, visual question answering, and image retrieval techniques that focus on using GNNs as the main part of their architecture.

20 pages, 5 figures, 5 tables

References in corpus (9)

• A Comprehensive Survey on Graph Neural Networks
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Knowledge Graphs
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• Aligning Linguistic Words and Visual Semantic Units for Image Captioning
• Gender and Racial Bias in Visual Question Answering Datasets
• Transforming Visual Scene Graphs to Image Captions
• A survey on VQA_Datasets and Approaches

Cited by in corpus (3)

• A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
• Attention-based transformer models for image captioning across languages: An in-depth survey and evaluation
• MAS-KCL: Knowledge component graph structure learning with large language model-based agentic workflow