CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Abstract

When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover shortcomings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Deep Residual Learning for Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Simple Baseline for Visual Question Answering
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Stacked Attention Networks for Image Question Answering
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Learning to Compose Neural Networks for Question Answering
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• Compact Bilinear Pooling
• Visual7W: Grounded Question Answering in Images
• Yin and Yang: Balancing and Answering Binary Visual Questions

Cited by in corpus (30)

• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Compositional Attention Networks for Machine Reasoning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• CHALET: Cornell House Agent Learning Environment
• FVQA: Fact-based Visual Question Answering
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• Language-Conditioned Graph Networks for Relational Reasoning
• DVQA: Understanding Data Visualizations via Question Answering
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer
• Scene Graph Generation via Conditional Random Fields
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Factor Graph Attention
• Visual Entailment Task for Visually-Grounded Language Learning
• Adversarial Attacks Beyond the Image Space
• Information Maximizing Visual Question Generation
• MarioQA: Answering Questions by Watching Gameplay Videos
• Better Set Representations For Relational Reasoning
• Learning to Disambiguate by Asking Discriminative Questions
• Grounding Spatio-Temporal Language with Transformers
• Embodied Language Grounding with 3D Visual Feature Representations
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Visual Reasoning with Natural Language
• Composing Text and Image for Image Retrieval - An Empirical Odyssey
• Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization
• Deep Algorithmic Question Answering: Towards a Compositionally Hybrid AI for Algorithmic Reasoning
• A new dataset and model for learning to understand navigational instructions
• Weakly-supervised multi-class object localization using only object counts as labels