Human-Adversarial Visual Question Answering
arXiv:2106.02280
Abstract
Performance on the most commonly used Visual Question Answering dataset (VQA v2) is starting to approach human accuracy. However, in interacting with state-of-the-art VQA models, it is clear that the problem is far from being solved. In order to stress test VQA models, we benchmark them against human-adversarial examples. Human subjects interact with a state-of-the-art VQA model, and for each image in the dataset, attempt to find a question where the model's predicted answer is incorrect. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples. We conduct an extensive analysis of the collected adversarial examples and provide guidance on future research directions. We hope that this Adversarial VQA (AdVQA) benchmark can help drive progress in the field and advance the state of the art.
arXiv comment22 pages, 13 figures. First two authors contributed equally
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Decoupled Weight Decay Regularization
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Rosetta: Large scale system for text detection and recognition in images
- Supervised Multimodal Bitransformers for Classifying Images and Text
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Interpretable Counting for Visual Question Answering
- UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- Quizbowl: The Case for Incremental Question Answering