Visual Question Answering: A Survey of Methods and Datasets

Abstract

Visual Question Answering (VQA) is a challenging task that has received increasing attention from both the computer vision and the natural language processing communities. Given an image and a question in natural language, it requires reasoning over visual elements of the image and general knowledge to infer the correct answer. In the first part of this survey, we examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to connect the visual and textual modalities. In particular, we examine the common approach of combining convolutional and recurrent neural networks to map images and questions to a common feature space. We also discuss memory-augmented and modular architectures that interface with structured knowledge bases. In the second part of this survey, we review the datasets available for training and evaluating VQA systems. The various datatsets contain questions at different levels of complexity, which require different capabilities and types of reasoning. We examine in depth the question/answer pairs from the Visual Genome project, and evaluate the relevance of the structured annotations of images with scene graphs for VQA. Finally, we discuss promising future directions for the field, in particular the connection to structured knowledge bases and the use of natural language processing models.

25 pages

References in corpus (33)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Dynamic Memory Networks for Visual and Textual Question Answering
• Large-scale Simple Question Answering with Memory Networks
• Verb Semantics and Lexical Selection
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Residual Learning for Visual QA
• Describing Videos by Exploiting Temporal Structure
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Learning to Compose Neural Networks for Question Answering
• Language to Logical Form with Neural Attention
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• A Joint Model of Language and Perception for Grounded Attribute Learning
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Towards Neural Network-based Reasoning
• Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
• FVQA: Fact-based Visual Question Answering
• Compositional Memory for Visual Question Answering
• Recurrent Neural Networks with External Memory for Language Understanding
• Revisiting Visual Question Answering Baselines
• Image Captioning and Visual Question Answering Based on Attributes and External Knowledge
• DualNet: Domain-Invariant Network for Visual Question Answering
• A Diagram Is Worth A Dozen Images

Cited by in corpus (14)

• Detecting Visual Relationships with Deep Relational Networks
• Zero-Shot Visual Question Answering
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• DVQA: Understanding Data Visualizations via Question Answering
• SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set
• An Analysis of Visual Question Answering Algorithms
• Survey of Visual Question Answering: Datasets and Techniques
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Towards a Framework for Visual Intelligence in Service Robotics: Epistemic Requirements and Gap Analysis
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Empirically Verifying Hypotheses Using Reinforcement Learning
• Recurrent and Contextual Models for Visual Question Answering