Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Abstract

Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing KVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the correct answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. Inspired by the human cognition theory, in this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. Thereinto, the visual graph and semantic graph are regarded as image-conditioned instantiation of the factual graph. On top of these new representations, we re-formulate Knowledge-based Visual Question Answering as a recurrent reasoning process for obtaining complementary evidence from multimodal information. To this end, we decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol ( GRUC ) module that conducts parallel reasoning over both visual and semantic information. By stacking the modules multiple times, our model performs transitive reasoning and obtains question-oriented concept representations under the constrain of different modalities. Finally, we perform graph neural networks to infer the global-optimal answer by jointly considering all the concepts. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA, and demonstrate the effectiveness and interpretability of our model with extensive experiments.

Accepted at Pattern Recognition. arXiv admin note: substantial text overlap with arXiv:2006.09073

References in corpus (10)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• LanczosNet: Multi-Scale Deep Graph Convolutional Networks
• Learning Conditioned Graph Structures for Interpretable Visual Question Answering
• Relation-Aware Graph Attention Network for Visual Question Answering
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

Cited by in corpus (8)

• Multi-Modal Knowledge Graph Construction and Application: A Survey
• Graph Neural Networks in Vision-Language Image Understanding: A Survey
• Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering
• Boon: A Neural Search Engine for Cross-Modal Information Retrieval
• Q4EDA: A Novel Strategy for Textual Information Retrieval Based on User Interactions with Visual Representations of Time Series