From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge
arXiv:1511.03292
Abstract
In this paper we propose the construction of linguistic descriptions of images. This is achieved through the extraction of scene description graphs (SDGs) from visual scenes using an automatically constructed knowledge base. SDGs are constructed using both vision and reasoning. Specifically, commonsense reasoning is applied on (a) detections obtained from existing perception methods on given images, (b) a "commonsense" knowledge base constructed using natural language processing of image annotations and (c) lexical ontological knowledge from resources such as WordNet. Amazon Mechanical Turk(AMT)-based evaluations on Flickr8k, Flickr30k and MS-COCO datasets show that in most cases, sentences auto-constructed from SDGs obtained by our method give a more relevant and thorough description of an image than a recent state-of-the-art image caption based approach. Our Image-Sentence Alignment Evaluation results are also comparable to that of the recent state-of-the art approaches.
References in corpus (3)
Cited by in corpus (11)
- A Comprehensive Survey of Scene Graphs: Generation and Application
- Neural Self Talk: Image Understanding via Continuous Questioning and Answering
- Object Relation Detection Based on One-shot Learning
- Re-evaluating Automatic Metrics for Image Captioning
- Relational Reasoning using Prior Knowledge for Visual Captioning
- A Review on Intelligent Object Perception Methods Combining Knowledge-based Reasoning and Machine Learning
- iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability
- TIGEr: Text-to-Image Grounding for Image Caption Evaluation
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- SPICE: Semantic Propositional Image Caption Evaluation
- REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning