C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset

Abstract

Visual Question Answering (VQA) has received a lot of attention over the past couple of years. A number of deep learning models have been proposed for this task. However, it has been shown that these models are heavily driven by superficial correlations in the training data and lack compositionality -- the ability to answer questions about unseen compositions of seen concepts. This compositionality is desirable and central to intelligence. In this paper, we propose a new setting for Visual Question Answering where the test question-answer pairs are compositionally novel compared to training question-answer pairs. To facilitate developing models under this setting, we present a new compositional split of the VQA v1.0 dataset, which we call Compositional VQA (C-VQA). We analyze the distribution of questions and answers in the C-VQA splits. Finally, we evaluate several existing VQA models under this new setting and show that the performances of these models degrade by a significant amount compared to the original VQA setting.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Zero-Shot Visual Question Answering
• Learning to generalize to new compositions in image understanding
• GuessWhat?! Visual object discovery through multi-modal dialogue
• An Empirical Evaluation of Visual Question Answering for Novel Objects

Cited by in corpus (18)

• A causal view of compositional zero-shot recognition
• Generative Models of Visually Grounded Imagination
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• DocVQA: A Dataset for VQA on Document Images
• Answer Them All! Toward Universal Visual Question Answering Models
• Zero-Shot Transfer VQA Dataset
• Zero-shot Visual Question Answering using Knowledge Graph
• EKTVQA: Generalized use of External Knowledge to empower Scene Text in Text-VQA
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Learning Associative Inference Using Fast Weight Memory
• Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• Generating Diverse Programs with Instruction Conditioned Reinforced Adversarial Learning
• What is Learned in Visually Grounded Neural Syntax Acquisition
• Understanding in Artificial Intelligence
• P $\approx$ NP, at least in Visual Question Answering
• On Controlled DeEntanglement for Natural Language Processing