Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research, and provide an open source implementation in TensorFlow.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1411.4555

References in corpus (14)

• Efficient Estimation of Word Representations in Vector Space
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Recurrent Neural Network Regularization
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Explain Images with Multimodal Recurrent Neural Networks
• Exploring Nearest Neighbor Approaches for Image Captioning
• Technical Report: Image Captioning with Semantically Similar Images

Cited by in corpus (128)

• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• Dual-Path Convolutional Image-Text Embeddings with Instance Loss
• Eigen-CAM: Class Activation Map using Principal Components
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
• Semantic Object Accuracy for Generative Text-to-Image Synthesis
• Bayesian Recurrent Neural Networks
• 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents
• SAFE: Similarity-Aware Multi-Modal Fake News Detection
• Natural Language Generation for Electronic Health Records
• Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Decoupled Novel Object Captioner
• Cutting Away the Confusion From Crowdtesting
• Context-Dependent Diffusion Network for Visual Relationship Detection
• Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media
• Supervised Contrastive Learning for Multimodal Unreliable News Detection in COVID-19 Pandemic
• Reconstruction Network for Video Captioning
• Accelerating Physics-Based Simulations Using Neural Network Proxies: An Application in Oil Reservoir Modeling
• Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving
• Comparative evaluation of CNN architectures for Image Caption Generation
• Gender and Racial Bias in Visual Question Answering Datasets
• 3G structure for image caption generation
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• Discrete and continuous representations and processing in deep learning: Looking forward
• MusCaps: Generating Captions for Music Audio
• Detecting Hateful Memes Using a Multimodal Deep Ensemble
• CgT-GAN: CLIP-guided Text GAN for Image Captioning
• Normalized and Geometry-Aware Self-Attention Network for Image Captioning
• Unpaired Image Captioning via Scene Graph Alignments
• Learning Semantic Concepts and Order for Image and Sentence Matching
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• SemStyle: Learning to Generate Stylised Image Captions using Unaligned Text
• Trading Off Diversity and Quality in Natural Language Generation
• Watch What You Just Said: Image Captioning with Text-Conditional Attention
• SLAM for Visually Impaired People: a Survey
• Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network
• Position Focused Attention Network for Image-Text Matching
• Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition
• Generating Diverse and Meaningful Captions
• "You Know What to Do": Proactive Detection of YouTube Videos Targeted by Coordinated Hate Attacks
• Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Multimodal Image Captioning for Marketing Analysis
• SIGAN: A Novel Image Generation Method for Solar Cell Defect Segmentation and Augmentation
• An Empirical Investigation into the Use of Image Captioning for Automated Software Documentation
• Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning
• Exploring the Applications of Faster R-CNN and Single-Shot Multi-box Detection in a Smart Nursery Domain
• TPsgtR: Neural-Symbolic Tensor Product Scene-Graph-Triplet Representation for Image Captioning
• Unpaired Image Captioning by Language Pivoting
• AutoCaption: Image Captioning with Neural Architecture Search
• Dissecting the Meme Magic: Understanding Indicators of Virality in Image Memes
• Multi-modal Discriminative Model for Vision-and-Language Navigation
• Multimodal Memory Modelling for Video Captioning
• Explore and Explain: Self-supervised Navigation and Recounting
• Semi-Supervised Panoptic Narrative Grounding
• Quantifying the amount of visual information used by neural caption generators
• Pre-gen metrics: Predicting caption quality metrics without generating captions
• Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation
• Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity
• Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning
• End-to-End Attention-based Image Captioning
• Better Text Understanding Through Image-To-Text Transfer
• Fast Sequence Generation with Multi-Agent Reinforcement Learning
• Improving Movement Predictions of Traffic Actors in Bird's-Eye View Models using GANs and Differentiable Trajectory Rasterization
• VISIR: Visual and Semantic Image Label Refinement
• Self-Segregating and Coordinated-Segregating Transformer for Focused Deep Multi-Modular Network for Visual Question Answering
• Joint Visual Grounding with Language Scene Graphs
• Batch-normalized Recurrent Highway Networks
• Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning
• On The Coherence of Quantitative Evaluation of Visual Explanations
• Query by Semantic Sketch
• Towards Overcoming False Positives in Visual Relationship Detection
• LUCSS: Language-based User-customized Colourization of Scene Sketches
• An Empirical Study of Language CNN for Image Captioning
• Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication
• SuperCaptioning: Image Captioning Using Two-dimensional Word Embedding
• Learning Dynamic Author Representations with Temporal Language Models
• Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning
• SACT: Self-Aware Multi-Space Feature Composition Transformer for Multinomial Attention for Video Captioning
• Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces
• Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioning
• A Deep Decoder Structure Based on WordEmbedding Regression for An Encoder-Decoder Based Model for Image Captioning
• Real-time Visual Object Tracking with Natural Language Description
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• Towards Unique and Informative Captioning of Images
• AIBench Scenario: Scenario-distilling AI Benchmarking
• Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
• Image Captioning with Clause-Focused Metrics in a Multi-Modal Setting for Marketing
• Improving Image Captioning by Leveraging Knowledge Graphs
• SuperOCR: A Conversion from Optical Character Recognition to Image Captioning
• FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptions
• CLIPScore: A Reference-free Evaluation Metric for Image Captioning
• Modularized Textual Grounding for Counterfactual Resilience
• ReFrame: Rectification Framework for Image Explaining Architectures
• Graphine: A Dataset for Graph-aware Terminology Definition Generation
• Symbolic music generation conditioned on continuous-valued emotions
• Towards Embodied Scene Description
• Recognizing and Curating Photo Albums via Event-Specific Image Importance
• There Once Was a Really Bad Poet, It Was Automated but You Didn't Know It
• Physical Cue based Depth-Sensing by Color Coding with Deaberration Network
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• Event Recognition with Automatic Album Detection based on Sequential Processing, Neural Attention and Image Captioning
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Model Asset eXchange: Path to Ubiquitous Deep Learning Deployment
• Introducing the DOME Activation Functions
• Caption Enriched Samples for Improving Hateful Memes Detection
• Sentence Semantic Regression for Text Generation
• Building Chatbots from Forum Data: Model Selection Using Question Answering Metrics
• Towards Instance Segmentation with Object Priority: Prominent Object Detection and Recognition
• Recurrent Models for Situation Recognition
• Learning Visual N-Grams from Web Data
• Integrating Deep Learning and Augmented Reality to Enhance Situational Awareness in Firefighting Environments
• Structured Pruning of Recurrent Neural Networks through Neuron Selection
• Semantic bottleneck for computer vision tasks
• Logic could be learned from images
• Shuffle-Then-Assemble: Learning Object-Agnostic Visual Relationship Features
• Scene-based Factored Attention for Image Captioning
• Sloth Search System at the Video Browser Showdown 2018 - Final Notes
• Better Understanding Hierarchical Visual Relationship for Image Caption
• Experimenting with Self-Supervision using Rotation Prediction for Image Captioning
• Towards ECDSA key derivation from deep embeddings for novel Blockchain applications
• Characterization and recognition of handwritten digits using Julia
• A Multi-Modal Approach to Infer Image Affect
• Image Captioning with Visual Object Representations Grounded in the Textual Modality