A Self-Explainable Stylish Image Captioning Framework via Multi-References

Abstract

In this paper, we propose to build a stylish image captioning model through a Multi-style Multi modality mechanism (2M). We demonstrate that with 2M, we can build an effective stylish captioner and that multi-references produced by the model can also support explaining the model through identifying erroneous input features on faulty examples. We show how this 2M mechanism can be used to build stylish captioning models and show how these models can be utilized to provide explanations of likely errors in the models.

arXiv admin note: substantial text overlap with arXiv:2103.11186 This paper is under consideration at Computer Vision and Image Understanding

References in corpus (3)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• YFCC100M: The New Data in Multimedia Research
• "Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention