FVQA: Fact-based Visual Question Answering

Abstract

Visual Question Answering (VQA) has attracted a lot of attention in both Computer Vision and Natural Language Processing communities, not least because it offers insight into the relationships between two important sources of information. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. It excludes questions which require common sense, or basic factual knowledge to answer, for example. Here we introduce FVQA, a VQA dataset which requires, and supports, much deeper reasoning. FVQA only contains questions which require external information to answer. We thus extend a conventional visual question answering dataset, which contains image-question-answerg triplets, through additional image-question-answer-supporting fact tuples. The supporting fact is represented as a structural triplet, such as <Cat,CapableOf,ClimbingTrees>. We evaluate several baseline models on the FVQA dataset, and describe a novel model which is capable of reasoning about an image on the basis of supporting facts.

arXiv comment16 pages

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Dynamic Memory Networks for Visual and Textual Question Answering
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Large-scale Simple Question Answering with Memory Networks
• Verb Semantics and Lexical Selection
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Question Answering with Subgraph Embeddings
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Open Question Answering with Weakly Supervised Embedding Models
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
• Uncovering Temporal Context for Video Question and Answering
• Compositional Memory for Visual Question Answering
• Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning
• Joint Video and Text Parsing for Understanding Events and Answering Queries

Cited by in corpus (13)

• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Visual Question Answering: A Survey of Methods and Datasets
• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs
• Scene Graph Generation with External Knowledge and Image Reconstruction
• MemexQA: Visual Memex Question Answering
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents
• Linguistically Driven Graph Capsule Network for Visual Question Reasoning
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Sequential Person Recognition in Photo Albums with a Recurrent Network
• Multilayer Dense Connections for Hierarchical Concept Classification