Unsupervised Cross-lingual Word Embedding by Multilingual Neural Language Models
arXiv:1809.02306
Abstract
We propose an unsupervised method to obtain cross-lingual embeddings without any parallel data or pre-trained word embeddings. The proposed model, which we call multilingual neural language models, takes sentences of multiple languages as an input. The proposed model contains bidirectional LSTMs that perform as forward and backward language models, and these networks are shared among all the languages. The other parameters, i.e. word embeddings and linear transformation between hidden states and outputs, are specific to each language. The shared LSTMs can capture the common sentence structure among all languages. Accordingly, word embeddings of each language are mapped into a common latent space, making it possible to measure the similarity of words across multiple languages. We evaluate the quality of the cross-lingual word embeddings on a word alignment task. Our experiments demonstrate that our model can obtain cross-lingual embeddings of much higher quality than existing unsupervised models when only a small amount of monolingual data (i.e. 50k sentences) are available, or the domains of monolingual data are different across languages.
arXiv comment8 pages
References in corpus (5)
- Efficient Estimation of Word Representations in Vector Space
- Word Translation Without Parallel Data
- Unsupervised Machine Translation Using Monolingual Corpora Only
- Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
- Sharing Network Parameters for Crosslingual Named Entity Recognition
Cited by in corpus (8)
- Cross-lingual Language Model Pretraining
- Multilingual Denoising Pre-training for Neural Machine Translation
- Cross-lingual Retrieval for Iterative Self-Supervised Training
- Pretrained Language Models for Text Generation: A Survey
- Traceability Support for Multi-Lingual Software Projects
- Sentence transition matrix: An efficient approach that preserves sentence semantics
- Lost in Evaluation: Misleading Benchmarks for Bilingual Dictionary Induction
- Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels