Quantifying the amount of visual information used by neural caption generators
arXiv:1810.05475 · doi:10.1007/978-3-030-11018-5_11
Abstract
This paper addresses the sensitivity of neural image caption generators to their visual input. A sensitivity analysis and omission analysis based on image foils is reported, showing that the extent to which image captioning architectures retain and are sensitive to visual information varies depending on the type of word being generated and the position in the caption as a whole. We motivate this work in the context of broader goals in the field to achieve more explainability in AI.
arXiv comment10 pages, 4 figures This publication will appear in the Proceedings of the First Workshop on Shortcomings in Vision and Language (2018). DOI to be inserted later
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Efficient Estimation of Word Representations in Vector Space
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Where to put the Image in an Image Caption Generator
- FOIL it! Find One mismatch between Image and Language caption