Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
arXiv:1711.06420
Abstract
Textual-visual cross-modal retrieval has been a hot research topic in both computer vision and natural language processing communities. Learning appropriate representations for multi-modal data is crucial for the cross-modal retrieval performance. Unlike existing image-text retrieval approaches that embed image-text pairs as single feature vectors in a common representational space, we propose to incorporate generative processes into the cross-modal feature embedding, through which we are able to learn not only the global abstract features but also the local grounded features. Extensive experiments show that our framework can well match images and sentences with complex content, and achieve the state-of-the-art cross-modal retrieval results on MSCOCO dataset.
10 pages, 6 figures, Accepted as spotlight at CVPR 2018
References in corpus (6)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Conditional Generative Adversarial Nets
- Generative Adversarial Text to Image Synthesis
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
Cited by in corpus (6)
- Show, Recall, and Tell: Image Captioning with Recall Mechanism
- Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
- Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching
- Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
- MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
- HUSE: Hierarchical Universal Semantic Embeddings