Identity-Aware Textual-Visual Matching with Latent Co-attention

Abstract

Textual-visual matching aims at measuring similarities between sentence descriptions and images. Most existing methods tackle this problem without effectively utilizing identity-level annotations. In this paper, we propose an identity-aware two-stage framework for the textual-visual matching problem. Our stage-1 CNN-LSTM network learns to embed cross-modal features with a novel Cross-Modal Cross-Entropy (CMCE) loss. The stage-1 network is able to efficiently screen easy incorrect matchings and also provide initial training point for the stage-2 training. The stage-2 CNN-LSTM network refines the matching results with a latent co-attention mechanism. The spatial attention relates each word with corresponding image regions while the latent semantic attention aligns different sentence structures to make the matching results more robust to sentence structure variations. Extensive experiments on three datasets with identity-level annotations show that our framework outperforms state-of-the-art approaches by large margins.

arXiv commentAccepted to ICCV 2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Going Deeper with Convolutions
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Explain Images with Multimodal Recurrent Neural Networks
• Simple Baseline for Visual Question Answering
• Person Search with Natural Language Description
• Dual Attention Networks for Multimodal Reasoning and Matching
• Visual7W: Grounded Question Answering in Images
• Person Re-identification in the Wild

Cited by in corpus (7)

• SCAN: Self-and-Collaborative Attention Network for Video Person Re-identification
• Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification
• Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association
• Adversarial Attribute-Image Person Re-identification
• Person Search Challenges and Solutions: A Survey