Learning Visual Knowledge Memory Networks for Visual Question Answering

Abstract

Visual question answering (VQA) requires joint comprehension of images and natural language questions, where many questions can't be directly or clearly answered from visual content but require reasoning from structured human knowledge with confirmation from visual content. This paper proposes visual knowledge memory network (VKMN) to address this issue, which seamlessly incorporates structured human knowledge and deep visual features into memory networks in an end-to-end learning framework. Comparing to existing methods for leveraging external knowledge for supporting VQA, this paper stresses more on two missing mechanisms. First is the mechanism for integrating visual contents with knowledge facts. VKMN handles this issue by embedding knowledge triples (subject, relation, target) and deep visual features jointly into the visual knowledge features. Second is the mechanism for handling multiple knowledge facts expanding from question and answer pairs. VKMN stores joint embedding using key-value pair structure in the memory networks so that it is easy to handle multiple facts. Experiments show that the proposed method achieves promising results on both VQA v1.0 and v2.0 benchmarks, while outperforms state-of-the-art methods on the knowledge-reasoning related questions.

arXiv commentSupplementary to CVPR 2018 version

References in corpus (25)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Reasoning about Entailment with Neural Attention
• Cascade R-CNN: Delving into High Quality Object Detection
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Attentive Pooling Networks
• Simple Baseline for Visual Question Answering
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Question Answering with Subgraph Embeddings
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• From Captions to Visual Concepts and Back
• ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Dual Attention Networks for Multimodal Reasoning and Matching
• Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• rnn : Recurrent Library for Torch
• Weakly Supervised Dense Video Captioning
• Structured Attentions for Visual Question Answering

Cited by in corpus (1)

• Improved RAMEN: Towards Domain Generalization for Visual Question Answering