Multilingual Multi-modal Embeddings for Natural Language Processing

Abstract

We propose a novel discriminative model that learns embeddings from multilingual and multi-modal data, meaning that our model can take advantage of images and descriptions in multiple languages to improve embedding quality. To that end, we introduce a modification of a pairwise contrastive estimation optimisation function as our training objective. We evaluate our embeddings on an image-sentence ranking (ISR), a semantic textual similarity (STS), and a neural machine translation (NMT) task. We find that the additional multilingual signals lead to improvements on both the ISR and STS tasks, and the discriminative cost can also be used in re-ranking $n$-best lists produced by NMT models, yielding strong improvements.

4 pages (5 including references), no figures

References in corpus (2)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Cited by in corpus (9)

• A Survey Of Cross-lingual Word Embedding Models
• Emergent Translation in Multi-Agent Communication
• Aligning Multilingual Word Embeddings for Cross-Modal Retrieval Task
• Learning Multilingual Word Embeddings Using Image-Text Data
• Towards Zero-shot Cross-lingual Image Retrieval and Tagging
• Lessons learned in multilingual grounded language learning
• MULE: Multimodal Universal Language Embedding
• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
• Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than Character Level