Show and Tell: A Neural Image Caption Generator

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. For instance, while the current state-of-the-art BLEU-1 score (the higher the better) on the Pascal dataset is 25, our approach yields 59, to be compared to human performance around 69. We also show BLEU-1 score improvements on Flickr30k, from 56 to 66, and on SBU, from 19 to 28. Lastly, on the newly released COCO dataset, we achieve a BLEU-4 of 27.7, which is the current state-of-the-art.

Cited by in corpus (104)

• VQA: Visual Question Answering
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Text Understanding from Scratch
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Fully Connected Deep Structured Networks
• Image Captioning with Semantic Attention
• Neural GPUs Learn Algorithms
• Neural Paraphrase Generation with Stacked Residual LSTM Networks
• Order Matters: Sequence to sequence for sets
• Using Descriptive Video Services to Create a Large Data Source for Video Annotation Research
• Multimodal Machine Learning: A Survey and Taxonomy
• Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection
• Language Models for Image Captioning: The Quirks and What Works
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Towards Diverse and Natural Image Descriptions via a Conditional GAN
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Natural Language Generation for Electronic Health Records
• The Devil is in the Middle: Exploiting Mid-level Representations for Cross-Domain Instance Matching
• Recurrent Network Models for Human Dynamics
• See, Hear, and Read: Deep Aligned Representations
• Long Short-Term Memory Over Tree Structures
• Recursive Recurrent Nets with Attention Modeling for OCR in the Wild
• Person Search with Natural Language Description
• Deep Reinforcement Learning-based Image Captioning with Embedding Reward
• CIDEr: Consensus-based Image Description Evaluation
• Multimodal Convolutional Neural Networks for Matching Image and Sentence
• Transferring Knowledge from a RNN to a DNN
• A Dataset for Movie Description
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Dual Attention Networks for Multimodal Reasoning and Matching
• Unsupervised Text Style Transfer using Language Models as Discriminators
• YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
• Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
• Generating Text with Deep Reinforcement Learning
• End-to-End Dense Video Captioning with Masked Transformer
• Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Reading Scene Text in Deep Convolutional Sequences
• A Neural Transducer
• Attribute Recognition by Joint Recurrent Learning of Context and Correlation
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
• Deep Recurrent Neural Networks for Acoustic Modelling
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Simple Image Description Generator via a Linear Phrase-Based Approach
• Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
• Weakly Supervised Dense Video Captioning
• Natural Language Object Retrieval
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• Context-aware Captions from Context-agnostic Supervision
• Learning with hidden variables
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• GuessWhat?! Visual object discovery through multi-modal dialogue
• Multi-Cue Zero-Shot Learning with Strong Supervision
• Learning Human Motion Models for Long-term Predictions
• Semantic Compositional Networks for Visual Captioning
• Scene Graph Generation via Conditional Random Fields
• Generating Descriptions with Grounded and Co-Referenced People
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Learning to Evaluate Image Captioning
• Deep Learning Methods for Efficient Large Scale Video Labeling
• Visual Learning of Arithmetic Operations
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• End-to-end Video-level Representation Learning for Action Recognition
• A Semi-supervised Framework for Image Captioning
• Sequence to Sequence Learning for Optical Character Recognition
• Ethical Artificial Intelligence
• An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning
• Recurrent Filter Learning for Visual Tracking
• Dense Captioning with Joint Inference and Visual Context
• Spatiotemporal Recurrent Convolutional Networks for Traffic Prediction in Transportation Networks
• Deep Quantization: Encoding Convolutional Activations with Deep Generative Model
• Top-down Visual Saliency Guided by Captions
• Large-scale Video Classification guided by Batch Normalized LSTM Translator
• Sketch2code: Generating a website from a paper mockup
• One Big Net For Everything
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Learning to Disambiguate by Asking Discriminative Questions
• DeepDiary: Automatic Caption Generation for Lifelogging Image Streams
• Spatial Memory for Context Reasoning in Object Detection
• Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing
• Examining Cooperation in Visual Dialog Models
• Disjoint Multi-task Learning between Heterogeneous Human-centric Tasks
• Few-Shot Object Recognition from Machine-Labeled Web Images
• EgoCoder: Intelligent Program Synthesis with Hierarchical Sequential Neural Network Model
• Multi-Task Spatiotemporal Neural Networks for Structured Surface Reconstruction
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Intrinsic Image Captioning Evaluation
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Zoom-RNN: A Novel Method for Person Recognition Using Recurrent Neural Networks
• Parallel Attention: A Unified Framework for Visual Object Discovery through Dialogs and Queries
• SPICE: Semantic Propositional Image Caption Evaluation
• Recurrent Models for Situation Recognition
• SentenceRacer: A Game with a Purpose for Image Sentence Annotation
• Mapping Natural Language Commands to Web Elements
• Generating Descriptions for Sequential Images with Local-Object Attention and Global Semantic Context Modelling
• A Sequential Set Generation Method for Predicting Set-Valued Outputs
• Composing Text and Image for Image Retrieval - An Empirical Odyssey