CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

References in corpus (1)

• Visual Madlibs: Fill in the blank Image Generation and Question Answering

Cited by in corpus (11)

• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Information Maximizing Visual Question Generation
• Learning to Disambiguate by Asking Discriminative Questions
• Grounding Spatio-Temporal Language with Transformers
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Visual Reasoning with Natural Language
• Composing Text and Image for Image Retrieval - An Empirical Odyssey
• Weakly-supervised multi-class object localization using only object counts as labels
• Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization
• Deep Algorithmic Question Answering: Towards a Compositionally Hybrid AI for Algorithmic Reasoning