Neural Attention for Image Captioning: Review of Outstanding Methods

Abstract

Image captioning is the task of automatically generating sentences that describe an input image in the best way possible. The most successful techniques for automatically generating image captions have recently used attentive deep learning models. There are variations in the way deep learning models with attention are designed. In this survey, we provide a review of literature related to attentive deep learning models for image captioning. Instead of offering a comprehensive review of all prior work on deep image captioning models, we explain various types of attention mechanisms used for the task of image captioning in deep learning models. The most successful deep learning models used for image captioning follow the encoder-decoder architecture, although there are differences in the way these models employ attention mechanisms. Via analysis on performance results from different attentive deep models for image captioning, we aim at finding the most successful types of attention mechanisms in deep models for image captioning. Soft attention, bottom-up attention, and multi-head attention are the types of attention mechanism widely used in state-of-the-art attentive deep learning models for image captioning. At the current time, the best results are achieved from variants of multi-head attention with bottom-up attention.

arXiv commentThis is the accepted version, which we are allowed to publish on arxiv based on Springer Nature policies. For the published version please refer to Springer Nature Artificial Intelligence Review Journal. DOI number is attached. For Citation refer to AIRE journal using DOI link

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Describing Multimedia Content using Attention-based Encoder--Decoder Networks
• Improving Image Captioning with Conditional Generative Adversarial Nets
• Show, Recall, and Tell: Image Captioning with Recall Mechanism
• Adaptively Aligned Image Captioning via Adaptive Attention Time