An Analysis of Visual Question Answering Algorithms

Abstract

In visual question answering (VQA), an algorithm must answer text-based questions about images. While multiple datasets for VQA have been created since late 2014, they all have flaws in both their content and the way algorithms are evaluated on them. As a result, evaluation scores are inflated and predominantly determined by answering easier questions, making it difficult to compare different methods. In this paper, we analyze existing VQA algorithms using a new dataset. It contains over 1.6 million questions organized into 12 different categories. We also introduce questions that are meaningless for a given image to force a VQA system to reason about image content. We propose new evaluation schemes that compensate for over-represented question-types and make it easier to study the strengths and weaknesses of algorithms. We analyze the performance of both baseline and state-of-the-art VQA models, including multi-modal compact bilinear pooling (MCB), neural module networks, and recurrent answering units. Our experiments establish how attention helps certain categories more than others, determine which models work better than others, and explain how simple models (e.g. MLP) can surpass more complex models (MCB) by simply learning to answer large, easy question categories.

arXiv commentTo appear in ICCV 2017. Visit http://kushalkafle.com/projects/tdiuc to download the dataset

References in corpus (24)

• Deep Residual Learning for Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Multimodal Residual Learning for Visual QA
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• A Focused Dynamic Attention Model for Visual Question Answering
• Learning to Compose Neural Networks for Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Training Recurrent Answering Units with Joint Loss Minimization for VQA
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Visual Question Answering: A Survey of Methods and Datasets
• Visual7W: Grounded Question Answering in Images
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Compositional Memory for Visual Question Answering
• Revisiting Visual Question Answering Baselines
• DualNet: Domain-Invariant Network for Visual Question Answering

Cited by in corpus (9)

• A simple neural network module for relational reasoning
• DVQA: Understanding Data Visualizations via Question Answering
• Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
• Overcoming Language Priors in Visual Question Answering with Adversarial Regularization
• Question Type Guided Attention in Visual Question Answering
• TallyQA: Answering Complex Counting Questions
• Cross-Dataset Adaptation for Visual Question Answering
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries