Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
arXiv:1910.03291 · doi:10.18653/v1/D19-6605
Abstract
In this paper, we propose a new approach to learn multimodal multilingual embeddings for matching images and their relevant captions in two languages. We combine two existing objective functions to make images and captions close in a joint embedding space while adapting the alignment of word embeddings between existing languages in our model. We show that our approach enables better generalization, achieving state-of-the-art performance in text-to-image and image-to-text retrieval task, and caption-caption similarity task. Two multimodal multilingual datasets are used for evaluation: Multi30k with German and English captions and Microsoft-COCO with English and Japanese captions.
References in corpus (6)
- Exploiting Similarities among Languages for Machine Translation
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
- Multimodal Pivots for Image Caption Translation
- Multilingual Multi-modal Embeddings for Natural Language Processing