Revisiting Cross Modal Retrieval
arXiv:1807.07364
Abstract
This paper proposes a cross-modal retrieval system that leverages on image and text encoding. Most multimodal architectures employ separate networks for each modality to capture the semantic relationship between them. However, in our work image-text encoding can achieve comparable results in terms of cross-modal retrieval without having to use a separate network for each modality. We show that text encodings can capture semantic relationships between multiple modalities. In our knowledge, this work is the first of its kind in terms of employing a single network and fused image-text embedding for cross-modal retrieval. We evaluate our approach on two famous multimodal datasets: MS-COCO and Flickr30K.
arXiv comment14 pages. Under review at ECCVW (MULA 2018)
References in corpus (10)
- Distributed Representations of Words and Phrases and their Compositionality
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- VoxCeleb2: Deep Speaker Recognition
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Gated Feedback Recurrent Neural Networks
- Representations of language in a model of visually grounded speech signal
- Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
- Learning Deep Representations of Medical Images using Siamese CNNs with Application to Content-Based Image Retrieval
- Image-Text Multi-Modal Representation Learning by Adversarial Backpropagation
- Visually Aligned Word Embeddings for Improving Zero-shot Learning