Microsoft COCO Captions: Data Collection and Evaluation Server

Abstract

In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

arXiv admin note: text overlap with arXiv:1411.4952

References in corpus (6)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Phrase-based Image Captioning
• Simple Image Description Generator via a Linear Phrase-Based Approach

Cited by in corpus (40)

• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Long Text Generation via Adversarial Training with Leaked Information
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• ImageNet pre-trained models with batch normalization
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
• Person Search with Natural Language Description
• Learning to Decode for Future Success
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• Visual Question Answering: A Survey of Methods and Datasets
• Spatio-Temporal Attention Models for Grounded Video Captioning
• Attention-Based Multimodal Fusion for Video Description
• Consensus-based Sequence Training for Video Captioning
• Semantic Compositional Networks for Visual Captioning
• Video Captioning with Transferred Semantic Attributes
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Referenceless Quality Estimation for Natural Language Generation
• Relational Reasoning using Prior Knowledge for Visual Captioning
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• A Survey on Biomedical Image Captioning
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning
• Multi-modal Discriminative Model for Vision-and-Language Navigation
• Survey of Recent Advances in Visual Question Answering
• Multimodal Memory Modelling for Video Captioning
• Generating captions without looking beyond objects
• Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering
• Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning
• Generating Diverse and Informative Natural Language Fashion Feedback
• M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
• Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories
• SPICE: Semantic Propositional Image Caption Evaluation
• Visual Reasoning with Natural Language
• Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data
• Learning Joint Representations of Videos and Sentences with Web Image Search
• The Long-Short Story of Movie Description
• Bidirectional Multirate Reconstruction for Temporal Modeling in Videos