AMC: Attention guided Multi-modal Correlation Learning for Image Search

Abstract

Given a user's query, traditional image search systems rank images according to its relevance to a single modality (e.g., image content or surrounding text). Nowadays, an increasing number of images on the Internet are available with associated meta data in rich modalities (e.g., titles, keywords, tags, etc.), which can be exploited for better similarity measure with queries. In this paper, we leverage visual and textual modalities for image search by learning their correlation with input query. According to the intent of query, attention mechanism can be introduced to adaptively balance the importance of different modalities. We propose a novel Attention guided Multi-modal Correlation (AMC) learning method which consists of a jointly learned hierarchy of intra and inter-attention networks. Conditioned on query's intent, intra-attention networks (i.e., visual intra-attention network and language intra-attention network) attend on informative parts within each modality; a multi-modal inter-attention network promotes the importance of the most query-relevant modalities. In experiments, we evaluate AMC models on the search logs from two real world image search engines and show a significant boost on the ranking of user-clicked images in search results. Additionally, we extend AMC models to caption ranking task on COCO dataset and achieve competitive results compared with recent state-of-the-arts.

CVPR 2017

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Recurrent Models of Visual Attention
• Deep Semantic Ranking Based Hashing for Multi-Label Image Retrieval
• Aligning where to see and what to tell: image caption with region-based attention and scene factorization

Cited by in corpus (5)

• Discriminative Feature Learning with Foreground Attention for Person Re-Identification
• Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
• Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
• Joint Visual-Textual Embedding for Multimodal Style Search
• Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention