Visual Question Answering Using Semantic Information from Image Descriptions
arXiv:2004.10966
Abstract
In this work, we propose a deep neural architecture that uses an attention mechanism which utilizes region based image features, the natural language question asked, and semantic knowledge extracted from the regions of an image to produce open-ended answers for questions asked in a visual question answering (VQA) task. The combination of both region based features and region based textual information about the image bolsters a model to more accurately respond to questions and potentially do so with less required training data. We evaluate our proposed architecture on a VQA task against a strong baseline and show that our method achieves excellent results on this task.
arXiv comment6 pages, 5 figures, The 34th International FLAIRS Conference
References in corpus (14)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- VQA: Visual Question Answering
- Image Captioning with Semantic Attention
- Learning to Count Objects in Natural Images for Visual Question Answering
- Stacked Attention Networks for Image Question Answering
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning
- In Defense of Grid Features for Visual Question Answering
- Improving Visual Question Answering by Referring to Generated Paragraph Captions
- Generating Question Relevant Captions to Aid Visual Question Answering