CNN+CNN: Convolutional Decoders for Image Captioning

Abstract

Image captioning is a challenging task that combines the field of computer vision and natural language processing. A variety of approaches have been proposed to achieve the goal of automatically describing an image, and recurrent neural network (RNN) or long-short term memory (LSTM) based models dominate this field. However, RNNs or LSTMs cannot be calculated in parallel and ignore the underlying hierarchical structure of a sentence. In this paper, we propose a framework that only employs convolutional neural networks (CNNs) to generate captions. Owing to parallel computing, our basic model is around 3 times faster than NIC (an LSTM-based model) during training time, while also providing better results. We conduct extensive experiments on MSCOCO and investigate the influence of the model width and depth. Compared with LSTM-based models that apply similar attention mechanisms, our proposed models achieves comparable scores of BLEU-1,2,3,4 and METEOR, and higher scores of CIDEr. We also test our model on the paragraph annotation dataset, and get higher CIDEr score compared with hierarchical LSTMs

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Natural Language Processing (almost) from Scratch
• WaveNet: A Generative Model for Raw Audio
• Convolutional Sequence to Sequence Learning
• Language Modeling with Gated Convolutional Networks
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Neural Machine Translation in Linear Time
• Image Captioning with Semantic Attention
• From Captions to Visual Concepts and Back
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Boosting Image Captioning with Attributes
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition

Cited by in corpus (13)

• Deep Learning Approaches on Image Captioning: A Review
• On Distinctive Image Captioning via Comparing and Reweighting
• Knowledge driven Description Synthesis for Floor Plan Interpretation
• Describing like humans: on diversity in image captioning
• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
• A Comprehensive Survey of Deep Learning for Image Captioning
• Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
• Gated Hierarchical Attention for Image Captioning
• Image Captioning based on Deep Learning Methods: A Survey
• Integrating Image Captioning with Rule-based Entity Masking
• Compare and Reweight: Distinctive Image Captioning Using Similar Images Sets
• Group-based Distinctive Image Captioning with Memory Attention
• Multi-Glimpse Network: A Robust and Efficient Classification Architecture based on Recurrent Downsampled Attention