Answering Questions about Data Visualizations using Efficient Bimodal Fusion
arXiv:1908.01801
Abstract
Chart question answering (CQA) is a newly proposed visual question answering (VQA) task where an algorithm must answer questions about data visualizations, e.g. bar charts, pie charts, and line graphs. CQA requires capabilities that natural-image VQA algorithms lack: fine-grained measurements, optical character recognition, and handling out-of-vocabulary words in both questions and answers. Without modifications, state-of-the-art VQA algorithms perform poorly on this task. Here, we propose a novel CQA algorithm called parallel recurrent fusion of image and language (PReFIL). PReFIL first learns bimodal embeddings by fusing question and image features and then intelligently aggregates these learned embeddings to answer the given question. Despite its simplicity, PReFIL greatly surpasses state-of-the art systems and human baselines on both the FigureQA and DVQA datasets. Additionally, we demonstrate that PReFIL can be used to reconstruct tables by asking a series of questions about a chart.
arXiv commentPresented at WACV, 2020
References in corpus (9)
- Understanding Neural Networks Through Deep Visualization
- VQA: Visual Question Answering
- Exploring Models and Data for Image Question Answering
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR