CIDEr: Consensus-based Image Description Evaluation
arXiv:1411.5726
Abstract
Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric (CIDEr) that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.
To appear in CVPR 2015
References in corpus (2)
Cited by in corpus (40)
- VQA: Visual Question Answering
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Exploring Nearest Neighbor Approaches for Image Captioning
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- Delving Deeper into Convolutional Networks for Learning Video Representations
- Towards Diverse and Natural Image Descriptions via a Conditional GAN
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- From Captions to Visual Concepts and Back
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning
- Discriminability objective for training descriptive captions
- Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
- Deep Reinforcement Learning-based Image Captioning with Embedding Reward
- Seeing with Humans: Gaze-Assisted Neural Image Captioning
- Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis
- SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
- Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
- Creativity: Generating Diverse Questions using Variational Autoencoders
- Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
- Weakly Supervised Dense Video Captioning
- Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
- Context-aware Captions from Context-agnostic Supervision
- Semantic Compositional Networks for Visual Captioning
- Consensus-based Sequence Training for Video Captioning
- Neural Self Talk: Image Understanding via Continuous Questioning and Answering
- Learning to Evaluate Image Captioning
- An Attempt towards Interpretable Audio-Visual Video Captioning
- Robustness Analysis of Visual QA Models by Basic Questions
- A Semi-supervised Framework for Image Captioning
- Attend and Interact: Higher-Order Object Interactions for Video Understanding
- Transform and Tell: Entity-Aware News Image Captioning
- Dense Captioning with Joint Inference and Visual Context
- Multimodal Dialogue State Tracking By QA Approach with Data Augmentation
- Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
- MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
- Grounded Video Description
- Recurrent Models for Situation Recognition
- Towards Diverse Paragraph Captioning for Untrimmed Videos
- AGenT Zero: Zero-shot Automatic Multiple-Choice Question Generation for Skill Assessments
- Bidirectional Multirate Reconstruction for Temporal Modeling in Videos