The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
arXiv:1809.04344
Abstract
We introduce MASSES, a simple evaluation metric for the task of Visual Question Answering (VQA). In its standard form, the VQA task is operationalized as follows: Given an image and an open-ended question in natural language, systems are required to provide a suitable answer. Currently, model performance is evaluated by means of a somehow simplistic metric: If the predicted answer is chosen by at least 3 human annotators out of 10, then it is 100% correct. Though intuitively valuable, this metric has some important limitations. First, it ignores whether the predicted answer is the one selected by the Majority (MA) of annotators. Second, it does not account for the quantitative Subjectivity (S) of the answers in the sample (and dataset). Third, information about the Semantic Similarity (SES) of the responses is completely neglected. Based on such limitations, we propose a multi-component metric that accounts for all these issues. We show that our metric is effective in providing a more fine-grained evaluation both on the quantitative and qualitative level.
arXiv comment10 pages, 7 figures
References in corpus (10)
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Exploring Nearest Neighbor Approaches for Image Captioning
- Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
- An Analysis of Visual Question Answering Algorithms
- Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
- Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets