Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering

Abstract

Accurately answering a question about a given image requires combining observations with general knowledge. While this is effortless for humans, reasoning with general knowledge remains an algorithmic challenge. To advance research in this direction a novel `fact-based' visual question answering (FVQA) task has been introduced recently along with a large set of curated facts which link two entities, i.e., two possible answers, via a relation. Given a question-image pair, deep network techniques have been employed to successively reduce the large set of facts until one of the two entities of the final remaining fact is predicted as the answer. We observe that a successive process which considers one fact at a time to form a local decision is sub-optimal. Instead, we develop an entity graph and use a graph convolutional network to `reason' about the correct answer by jointly considering all entities. We show on the challenging FVQA dataset that this leads to an improvement in accuracy of around 7% compared to the state of the art.

arXiv commentAccepted to NIPS 2018

References in corpus (23)

• Semi-Supervised Classification with Graph Convolutional Networks
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Dynamic Memory Networks for Visual and Textual Question Answering
• Large-scale Simple Question Answering with Memory Networks
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Residual Learning for Visual QA
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Open Question Answering with Weakly Supervised Embedding Models
• Learning Dependency-Based Compositional Semantics
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Question Answering over Knowledge Base with Neural Attention Combining Global Knowledge Information
• Compositional Memory for Visual Question Answering
• Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning
• Creativity: Generating Diverse Questions using Variational Autoencoders
• Visual Question Generation as Dual Task of Visual Question Answering
• Image Captioning and Visual Question Answering Based on Attributes and External Knowledge

Cited by in corpus (5)

• Passage Retrieval for Outside-Knowledge Visual Question Answering
• Multiwave COVID-19 Prediction from Social Awareness using Web Search and Mobility Data
• RSG-Net: Towards Rich Sematic Relationship Prediction for Intelligent Vehicle in Complex Environments
• Learning Global and Local Consistent Representations for Unsupervised Image Retrieval via Deep Graph Diffusion Networks
• Split and Connect: A Universal Tracklet Booster for Multi-Object Tracking