FigureQA: An Annotated Figure Dataset for Visual Reasoning

Abstract

We introduce FigureQA, a visual reasoning corpus of over one million question-answer pairs grounded in over 100,000 images. The images are synthetic, scientific-style figures from five classes: line plots, dot-line plots, vertical and horizontal bar graphs, and pie charts. We formulate our reasoning task by generating questions from 15 templates; questions concern various relationships between plot elements and examine characteristics like the maximum, the minimum, area-under-the-curve, smoothness, and intersection. To resolve, such questions often require reference to multiple plot elements and synthesis of information distributed spatially throughout a figure. To facilitate the training of machine learning systems, the corpus also includes side data that can be used to formulate auxiliary objectives. In particular, we provide the numerical data used to generate each figure as well as bounding-box annotations for all plot elements. We study the proposed visual reasoning task by training several models, including the recently proposed Relation Network as a strong baseline. Preliminary results indicate that the task poses a significant machine learning challenge. We envision FigureQA as a first step towards developing models that can intuitively recognize patterns from visual representations of data.

workshop paper at ICLR 2018

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A simple neural network module for relational reasoning
• FiLM: Visual Reasoning with a General Conditioning Layer
• Imagination improves Multimodal Translation
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• MarioQA: Answering Questions by Watching Gameplay Videos
• The "something something" video database for learning and evaluating visual common sense
• A Data Driven Approach for Compound Figure Separation Using Convolutional Neural Networks

Cited by in corpus (26)

• Towards Natural Language Interfaces for Data Visualization: A Survey
• A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Data Extraction from Charts via Single Deep Neural Network
• DVQA: Understanding Data Visualizations via Question Answering
• A Dataset of Alt Texts from HCI Publications: Analyses and Uses Towards Producing More Descriptive Alt Texts of Data Visualizations in Scientific Papers
• Document Collection Visual Question Answering
• A Review of Emerging Research Directions in Abstract Visual Reasoning
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• Reviving Static Charts into Live Charts
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• Structured Multimodal Attentions for TextVQA
• IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
• DocVQA: A Dataset for VQA on Document Images
• VisRecall: Quantifying Information Visualisation Recallability via Question Answering
• Communicating Visualizations without Visuals: Investigation of Visualization Alternative Text for People with Visual Impairments
• Figure Captioning with Reasoning and Sequence-Level Training
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• VisualMRC: Machine Reading Comprehension on Document Images
• Answering Questions about Data Visualizations using Efficient Bimodal Fusion
• Towards Solving Multimodal Comprehension
• AutoChart: A Dataset for Chart-to-Text Generation Task
• CHARTER: heatmap-based multi-type chart data extraction
• ICDAR 2021 Competition on Document VisualQuestion Answering
• Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network
• AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization