Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO

Abstract

By supporting multi-modal retrieval training and evaluation, image captioning datasets have spurred remarkable progress on representation learning. Unfortunately, datasets have limited cross-modal associations: images are not paired with other images, captions are only paired with other captions of the same image, there are no negative associations and there are missing positive cross-modal associations. This undermines research into how inter-modality learning impacts intra-modality tasks. We address this gap with Crisscrossed Captions (CxC), an extension of the MS-COCO dataset with human semantic similarity judgments for 267,095 intra- and inter-modality pairs. We report baseline results on CxC for strong existing unimodal and multimodal models. We also evaluate a multitask dual encoder trained on both image-caption and caption-caption pairs that crucially demonstrates CxC's value for measuring the influence of intra- and inter-modality learning.

arXiv commentTo be presented at EACL2021

References in corpus (3)

• Big Self-Supervised Models are Strong Semi-Supervised Learners
• A Deep Generative Framework for Paraphrase Generation
• End-to-End Retrieval in Continuous Space

Cited by in corpus (7)

• Adversarial Text-to-Image Synthesis: A Review
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
• Talk, Don't Write: A Study of Direct Speech-Based Image Retrieval
• Text-to-Image Generation Grounded by Fine-Grained User Attention
• PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling