CAPTION: Correction by Analyses, POS-Tagging and Interpretation of Objects using only Nouns
arXiv:2010.00839
Abstract
Recently, Deep Learning (DL) methods have shown an excellent performance in image captioning and visual question answering. However, despite their performance, DL methods do not learn the semantics of the words that are being used to describe a scene, making it difficult to spot incorrect words used in captions or to interchange words that have similar meanings. This work proposes a combination of DL methods for object detection and natural language processing to validate image's captions. We test our method in the FOIL-COCO data set, since it provides correct and incorrect captions for various images using only objects represented in the MS-COCO image data set. Results show that our method has a good overall performance, in some cases similar to the human performance.
arXiv commentPublished at the First Annual International Workshop on Interpretability: Methodologies and algorithms (IMA 2019)
References in corpus (5)
- VQA: Visual Question Answering
- The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Pre-gen metrics: Predicting caption quality metrics without generating captions
- Quantifying the amount of visual information used by neural caption generators