Figure Captioning with Reasoning and Sequence-Level Training
arXiv:1906.02850
Abstract
Figures, such as bar charts, pie charts, and line plots, are widely used to convey important information in a concise format. They are usually human-friendly but difficult for computers to process automatically. In this work, we investigate the problem of figure captioning where the goal is to automatically generate a natural language description of the figure. While natural image captioning has been studied extensively, figure captioning has received relatively little attention and remains a challenging problem. First, we introduce a new dataset for figure captioning, FigCAP, based on FigureQA. Second, we propose two novel attention mechanisms. To achieve accurate generation of labels in figures, we propose Label Maps Attention. To model the relations between figure labels, we propose Relation Maps Attention. Third, we use sequence-level training with reinforcement learning in order to directly optimizes evaluation metrics, which alleviates the exposure bias issue and further improves the models in generating long captions. Extensive experiments show that the proposed method outperforms the baselines, thus demonstrating a significant potential for the automatic captioning of vast repositories of figures.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- VQA: Visual Question Answering
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Simple Baseline for Visual Question Answering
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- FigureQA: An Annotated Figure Dataset for Visual Reasoning
- A Focused Dynamic Attention Model for Visual Question Answering
- Guiding Long-Short Term Memory for Image Caption Generation
- Simple Image Description Generator via a Linear Phrase-Based Approach