A Comprehensive Survey of Deep Learning for Image Captioning

Abstract

Generating a description of an image is called image captioning. Image captioning requires to recognize the important objects, their attributes and their relationships in an image. It also needs to generate syntactically and semantically correct sentences. Deep learning-based techniques are capable of handling the complexities and challenges of image captioning. In this survey paper, we aim to present a comprehensive review of existing deep learning-based image captioning techniques. We discuss the foundation of the techniques to analyze their performances, strengths and limitations. We also discuss the datasets and the evaluation metrics popularly used in deep learning based automatic image captioning.

arXiv comment36 Pages, Accepted as a Journal Paper in ACM Computing Surveys (October 2018)

References in corpus (40)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Categorical Reparameterization with Gumbel-Softmax
• Convolutional Sequence to Sequence Learning
• Generative Adversarial Text to Image Synthesis
• Conditional Image Generation with PixelCNN Decoders
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Exploring the Limits of Language Modeling
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Improved Image Captioning via Policy Gradient optimization of SPIDEr
• Explain Images with Multimodal Recurrent Neural Networks
• Image Captioning with Semantic Attention
• Language Models for Image Captioning: The Quirks and What Works
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Actor-Critic Sequence Training for Image Captioning
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• CNN+CNN: Convolutional Decoders for Image Captioning
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• A Convolutional Encoder Model for Neural Machine Translation
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• Rich Image Captioning in the Wild
• What value do explicit high level concepts have in vision to language problems?
• Learning Attributes Equals Multi-Source Domain Generalization
• Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects
• Text to Image Synthesis Using Generative Adversarial Networks
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Image Captioning with Deep Bidirectional LSTMs
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Multi-Instance Visual-Semantic Embedding
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
• Automatic Concept Discovery from Parallel Text and Visual Corpora
• Dense Captioning with Joint Inference and Visual Context

Cited by in corpus (4)

• Facebook Ad Engagement in the Russian Active Measures Campaign of 2016
• Multi-Image Summarization: Textual Summary from a Set of Cohesive Images
• Better Understanding Hierarchical Visual Relationship for Image Caption
• Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention