Joint Image Captioning and Question Answering
arXiv:1805.08389
Abstract
Answering visual questions need acquire daily common knowledge and model the semantic connection among different parts in images, which is too difficult for VQA systems to learn from images with the only supervision from answers. Meanwhile, image captioning systems with beam search strategy tend to generate similar captions and fail to diversely describe images. To address the aforementioned issues, we present a system to have these two tasks compensate with each other, which is capable of jointly producing image captions and answering visual questions. In particular, we utilize question and image features to generate question-related captions and use the generated captions as additional features to provide new knowledge to the VQA system. For image captioning, our system attains more informative results in term of the relative improvements on VQA tasks as well as competitive results using automated metrics. Applying our system to the VQA tasks, our results on VQA v2 dataset achieve 65.8% using generated captions and 69.1% using annotated captions in validation set and 68.4% in the test-standard set. Further, an ensemble of 10 models results in 69.7% in the test-standard split.
References in corpus (11)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VQA: Visual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Attention on Attention: Architectures for Visual Question Answering (VQA)
- VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
- Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
- Dynamic Filtering with Large Sampling Field for ConvNets