Show, Recall, and Tell: Image Captioning with Recall Mechanism

Abstract

Generating natural and accurate descriptions in image cap-tioning has always been a challenge. In this paper, we pro-pose a novel recall mechanism to imitate the way human con-duct captioning. There are three parts in our recall mecha-nism : recall unit, semantic guide (SG) and recalled-wordslot (RWS). Recall unit is a text-retrieval module designedto retrieve recalled words for images. SG and RWS are de-signed for the best use of recalled words. SG branch cangenerate a recalled context, which can guide the process ofgenerating caption. RWS branch is responsible for copyingrecalled words to the caption. Inspired by pointing mecha-nism in text summarization, we adopt a soft switch to balancethe generated-word probabilities between SG and RWS. Inthe CIDEr optimization step, we also introduce an individualrecalled-word reward (WR) to boost training. Our proposedmethods (SG+RWS+WR) achieve BLEU-4 / CIDEr / SPICEscores of 36.6 / 116.9 / 21.3 with cross-entropy loss and 38.7 /129.1 / 22.4 with CIDEr optimization on MSCOCO Karpathytest split, which surpass the results of other state-of-the-artmethods.

arXiv commentPublished in AAAI 2020

References in corpus (8)

• Efficient Estimation of Word Representations in Vector Space
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Discriminability objective for training descriptive captions
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

Cited by in corpus (8)

• Neural Attention for Image Captioning: Review of Outstanding Methods
• Transforming Visual Scene Graphs to Image Captions
• Dual-Level Collaborative Transformer for Image Captioning
• Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
• A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes
• Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network
• Macroscopic Control of Text Generation for Image Captioning
• Geometry-Entangled Visual Semantic Transformer for Image Captioning