Microsoft COCO Captions: Data Collection and Evaluation Server

Abstract

In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

arXiv admin note: text overlap with arXiv:1411.4952

References in corpus (6)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Explain Images with Multimodal Recurrent Neural Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Phrase-based Image Captioning
• Simple Image Description Generator via a Linear Phrase-Based Approach

Cited by in corpus (67)

• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Multi-Task Learning with Deep Neural Networks: A Survey
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Long Text Generation via Adversarial Training with Leaked Information
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• ImageNet pre-trained models with batch normalization
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization
• Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection
• Recurrent Topic-Transition GAN for Visual Paragraph Generation
• Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space
• Person Search with Natural Language Description
• Learning to Decode for Future Success
• Seeing with Humans: Gaze-Assisted Neural Image Captioning
• Visual Question Answering: A Survey of Methods and Datasets
• Spatio-Temporal Attention Models for Grounded Video Captioning
• Attention-Based Multimodal Fusion for Video Description
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• Semantic Compositional Networks for Visual Captioning
• Consensus-based Sequence Training for Video Captioning
• Video Captioning with Transferred Semantic Attributes
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Generating Diverse and Meaningful Captions
• Referenceless Quality Estimation for Natural Language Generation
• Relational Reasoning using Prior Knowledge for Visual Captioning
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• Spatio-Temporal Dynamics and Semantic Attribute Enriched Visual Encoding for Video Captioning
• A Survey on Biomedical Image Captioning
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• Neuroevolution of Neural Network Architectures Using CoDeepNEAT and Keras
• Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning
• Information Maximizing Visual Question Generation
• Multi-modal Discriminative Model for Vision-and-Language Navigation
• Survey of Recent Advances in Visual Question Answering
• Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models
• Multimodal Memory Modelling for Video Captioning
• Generating captions without looking beyond objects
• All-in-One Image-Grounded Conversational Agents
• Multimodal Generative Models for Compositional Representation Learning
• Improving Question Generation With to the Point Context
• Learning Object Detection from Captions via Textual Scene Attributes
• Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering
• Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning
• In-Home Daily-Life Captioning Using Radio Signals
• Semantic-aware Image Deblurring
• CoTK: An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
• Generating Diverse and Informative Natural Language Fashion Feedback
• Ladder Loss for Coherent Visual-Semantic Embedding
• Structure-Aware Generation Network for Recipe Generation from Images
• M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention
• Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream
• Empirical Autopsy of Deep Video Captioning Frameworks
• Improving Image Captioning by Leveraging Knowledge Graphs
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories
• Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data
• Visual Reasoning with Natural Language
• SPICE: Semantic Propositional Image Caption Evaluation
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Video Captioning Using Weak Annotation
• Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
• The Long-Short Story of Movie Description
• Bidirectional Multirate Reconstruction for Temporal Modeling in Videos
• Learning Joint Representations of Videos and Sentences with Web Image Search
• Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder