Watch What You Just Said: Image Captioning with Text-Conditional Attention

Abstract

Attention mechanisms have attracted considerable interest in image captioning due to its powerful performance. However, existing methods use only visual content as attention and whether textual context can improve attention in image captioning remains unsolved. To explore this problem, we propose a novel attention mechanism, called \textit{text-conditional attention}, which allows the caption generator to focus on certain image features given previously generated text. To obtain text-related image features for our attention model, we adopt the guiding Long Short-Term Memory (gLSTM) captioning architecture with CNN fine-tuning. Our proposed method allows joint learning of the image embedding, text embedding, text-conditional attention and language model with one network architecture in an end-to-end manner. We perform extensive experiments on the MS-COCO dataset. The experimental results show that our method outperforms state-of-the-art captioning methods on various quantitative metrics as well as in human evaluation, which supports the use of our text-conditional attention in image captioning.

source code is available online

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• LSTM: A Search Space Odyssey
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Image Captioning with Semantic Attention

Cited by in corpus (6)

• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• End-to-End Dense Video Captioning with Masked Transformer
• Boosting Image Captioning with Attributes
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• Image Based Review Text Generation with Emotional Guidance
• Grounded Video Description