Improving Image Captioning by Leveraging Knowledge Graphs
arXiv:1901.08942
Abstract
We explore the use of a knowledge graphs, that capture general or commonsense knowledge, to augment the information extracted from images by the state-of-the-art methods for image captioning. The results of our experiments, on several benchmark data sets such as MS COCO, as measured by CIDEr-D, a performance metric for image captioning, show that the variants of the state-of-the-art methods for image captioning that make use of the information extracted from knowledge graphs can substantially outperform those that rely solely on the information extracted from images.
Accepted by WACV'19
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Explain Images with Multimodal Recurrent Neural Networks
- Exploring Nearest Neighbor Approaches for Image Captioning