Pre-gen metrics: Predicting caption quality metrics without generating captions

Abstract

Image caption generation systems are typically evaluated against reference outputs. We show that it is possible to predict output quality without generating the captions, based on the probability assigned by the neural model to the reference captions. Such pre-gen metrics are strongly correlated to standard evaluation metrics.

arXiv comment13 pages, 6 figures This publication will appear in the Proceedings of the First Workshop on Shortcomings in Vision and Language (2018). DOI to be inserted later

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Where to put the Image in an Image Caption Generator
• FOIL it! Find One mismatch between Image and Language caption

Cited by in corpus (3)

• CAPTION: Correction by Analyses, POS-Tagging and Interpretation of Objects using only Nouns
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Visuallly Grounded Generation of Entailments from Premises