Learning Multilingual Word Embeddings Using Image-Text Data

Abstract

There has been significant interest recently in learning multilingual word embeddings -- in which semantically similar words across languages have similar embeddings. State-of-the-art approaches have relied on expensive labeled data, which is unavailable for low-resource languages, or have involved post-hoc unification of monolingual embeddings. In the present paper, we investigate the efficacy of multilingual embeddings learned from weakly-supervised image-text data. In particular, we propose methods for learning multilingual embeddings using image-text data, by enforcing similarity between the representations of the image and that of the text. Our experiments reveal that even without using any expensive labeled data, a bag-of-words-based embedding model trained on image-text data achieves performance comparable to the state-of-the-art on crosslingual semantic similarity tasks.

References in corpus (7)

• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Exploiting Similarities among Languages for Machine Translation
• Massively Multilingual Word Embeddings
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Graph-RISE: Graph-Regularized Image Semantic Embedding
• Multilingual Multi-modal Embeddings for Natural Language Processing

Cited by in corpus (2)

• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• Practical Comparable Data Collection for Low-Resource Languages via Images