Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

Abstract

Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision. Tasks such as text-based image retrieval and image captioning were designed to test this ability but come with evaluation measures that have a high variance or are difficult to interpret. We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images. The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates the reliability problems of retrieval evaluations, and focuses on the system's ability to understand fine-grained visual structure. We gather a BISON dataset that complements the COCO dataset and use it to evaluate modern text-based image retrieval and image captioning systems. Our results provide novel insights into the performance of these systems. The COCO-BISON dataset and corresponding evaluation code are publicly available from \url{http://hexianghu.com/bison/}.

References in corpus (14)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Deep Residual Learning for Image Recognition
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• VQA: Visual Question Answering
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Visual Storytelling
• Learning to Ask: Neural Question Generation for Reading Comprehension
• Discriminability objective for training descriptive captions
• Modeling Context in Referring Expressions
• Reasoning About Pragmatics with Neural Listeners and Speakers
• Context-aware Captions from Context-agnostic Supervision

Cited by in corpus (2)

• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension