CLIPScore: A Reference-free Evaluation Metric for Image Captioning
arXiv:2104.08718
Abstract
Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality. In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPScore, achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE. Information gain experiments demonstrate that CLIPScore, with its tight focus on image-text compatibility, is complementary to existing reference-based metrics that emphasize text-text similarities. Thus, we also present a reference-augmented version, RefCLIPScore, which achieves even higher correlation. Beyond literal description tasks, several case studies reveal domains where CLIPScore performs well (clip-art images, alt-text rating), but also where it is relatively weaker in comparison to reference-based metrics, e.g., news captions that require richer contextual knowledge.
References in corpus (8)
- BERTScore: Evaluating Text Generation with BERT
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
- Contrastive Learning for Image Captioning
- NUBIA: NeUral Based Interchangeability Assessor for Text Generation
- Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications