Revisiting Visual Question Answering Baselines

Abstract

Visual question answering (VQA) is an interesting learning setting for evaluating the abilities and shortcomings of current systems for image understanding. Many of the recently proposed VQA systems include attention or memory mechanisms designed to support "reasoning". For multiple-choice VQA, nearly all of these systems train a multi-class classifier on image and question features to predict an answer. This paper questions the value of these common practices and develops a simple alternative model based on binary classification. Instead of treating answers as competing choices, our model receives the answer as input and predicts whether or not an image-question-answer triplet is correct. We evaluate our model on the Visual7W Telling and the VQA Real Multiple Choice tasks, and find that even simple versions of our model perform competitively. Our best model achieves state-of-the-art performance on the Visual7W Telling task and compares surprisingly well with the most complex systems proposed for the VQA Real Multiple Choice task. We explore variants of the model and study its transferability between both datasets. We also present an error analysis of our model that suggests a key problem of current VQA systems lies in the lack of visual grounding of concepts that occur in the questions and answers. Overall, our results suggest that the performance of current VQA systems is not significantly better than that of systems designed to exploit dataset biases.

European Conference on Computer Vision

References in corpus (4)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Multimodal Residual Learning for Visual QA
• Visual Madlibs: Fill in the blank Image Generation and Question Answering

Cited by in corpus (14)

• Imagination improves Multimodal Translation
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Zero-Shot Visual Question Answering
• ShapeWorld - A new test methodology for multimodal language understanding
• Visual Question Answering: A Survey of Methods and Datasets
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• An Analysis of Visual Question Answering Algorithms
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity
• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
• Lessons learned in multilingual grounded language learning
• Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features