Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

Abstract

Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge.

References in corpus (17)

• Deep Residual Learning for Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Large-scale Simple Question Answering with Memory Networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Visual7W: Grounded Question Answering in Images
• Visual Question Answering with Memory-Augmented Networks

Cited by in corpus (13)

• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• Multi-modal Deep Analysis for Multimedia
• Relation-Aware Graph Attention Network for Visual Question Answering
• Scene Graph Generation with External Knowledge and Image Reconstruction
• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering
• A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
• CRIC: A VQA Dataset for Compositional Reasoning on Vision and Commonsense
• Zero-Shot Scene Graph Relation Prediction through Commonsense Knowledge Integration
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Saliency Prediction with External Knowledge