Stacked Cross Attention for Image-Text Matching

Abstract

In this paper, we study the problem of image-text matching. Inferring the latent semantic alignment between objects or other salient stuff (e.g. snow, sky, lawn) and the corresponding words in sentences allows to capture fine-grained interplay between vision and language, and makes image-text matching more interpretable. Prior work either simply aggregates the similarity of all possible pairs of regions and words without attending differentially to more and less important words or regions, or uses a multi-step attentional process to capture limited number of semantic alignments which is less interpretable. In this paper, we present Stacked Cross Attention to discover the full latent alignments using both image regions and words in a sentence as context and infer image-text similarity. Our approach achieves the state-of-the-art results on the MS-COCO and Flickr30K datasets. On Flickr30K, our approach outperforms the current best methods by 22.1% relatively in text retrieval from image query, and 18.2% relatively in image retrieval with text query (based on Recall@1). On MS-COCO, our approach improves sentence retrieval by 17.8% relatively and image retrieval by 16.6% relatively (based on Recall@1 using the 5K test set). Code has been made available at: https://github.com/kuanghuei/SCAN.

Accepted to ECCV 2018

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• A Hierarchical Neural Autoencoder for Paragraphs and Documents
• Language Models for Image Captioning: The Quirks and What Works
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning
• CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise
• Learning Semantic Concepts and Order for Image and Sentence Matching

Cited by in corpus (6)

• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• All-in-One Image-Grounded Conversational Agents
• Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization