An Empirical Study on Leveraging Scene Graphs for Visual Question Answering

Abstract

Visual question answering (Visual QA) has attracted significant attention these years. While a variety of algorithms have been proposed, most of them are built upon different combinations of image and language features as well as multi-modal attention and fusion. In this paper, we investigate an alternative approach inspired by conventional QA systems that operate on knowledge graphs. Specifically, we investigate the use of scene graphs derived from images for Visual QA: an image is abstractly represented by a graph with nodes corresponding to object entities and edges to object relationships. We adapt the recently proposed graph network (GN) to encode the scene graph and perform structured reasoning according to the input question. Our empirical studies demonstrate that scene graphs can already capture essential information of images and graph networks have the potential to outperform state-of-the-art Visual QA algorithms but with a much cleaner architecture. By analyzing the features generated by GNs we can further interpret the reasoning process, suggesting a promising direction towards explainable Visual QA.

arXiv commentAccepted as oral presentation at BMVC 2019

References in corpus (18)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• A simple neural network module for relational reasoning
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding
• Pixels to Graphs by Associative Embedding
• The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
• Scene Graph Generation by Iterative Message Passing
• Relation-Aware Graph Attention Network for Visual Question Answering
• Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation
• Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Referring Relationships
• VrR-VG: Refocusing Visually-Relevant Relationships

Cited by in corpus (7)

• A Comprehensive Survey of Scene Graphs: Generation and Application
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Graph Density-Aware Losses for Novel Compositions in Scene Graph Generation
• Bridging Knowledge Graphs to Generate Scene Graphs
• LRTA: A Transparent Neural-Symbolic Reasoning Framework with Modular Supervision for Visual Question Answering
• Attention Guided Semantic Relationship Parsing for Visual Question Answering
• Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering