Robust Cross-lingual Embeddings from Parallel Sentences

Abstract

Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation. However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to hold in practice (Søgaard et al., 2018), and fundamentally limits their performance. This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective. We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations. Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches while maintaining parity with the current state-of-the-art methods on word-translation. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference. As an additional advantage, our bilingual method leads to a much more pronounced improvement in the the quality of monolingual word vectors compared to other competing methods.

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Cross-lingual Language Model Pretraining
• Exploiting Similarities among Languages for Machine Translation
• Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features
• A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• An Autoencoder Approach to Learning Bilingual Word Representations
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Analyzing the Limitations of Cross-lingual Word Embedding Mappings
• Non-Adversarial Unsupervised Word Translation
• Crosslingual Document Embedding as Reduced-Rank Ridge Regression

Cited by in corpus (2)

• Leveraging Advantages of Interactive and Non-Interactive Models for Vector-Based Cross-Lingual Information Retrieval
• EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning