An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification

Abstract

End-to-end neural machine translation has overtaken statistical machine translation in terms of translation quality for some language pairs, specially those with large amounts of parallel data. Besides this palpable improvement, neural networks provide several new properties. A single system can be trained to translate between many languages at almost no additional cost other than training time. Furthermore, internal representations learned by the network serve as a new semantic representation of words -or sentences- which, unlike standard word embeddings, are learned in an essentially bilingual or even multilingual context. In view of these properties, the contribution of the present work is two-fold. First, we systematically study the NMT context vectors, i.e. output of the encoder, and their power as an interlingua representation of a sentence. We assess their quality and effectiveness by measuring similarities across translations, as well as semantically related and semantically unrelated sentence pairs. Second, as extrinsic evaluation of the first point, we identify parallel sentences in comparable corpora, obtaining an F1=98.2% on data from a shared task when using only NMT context vectors. Using context vectors jointly with similarity measures F1 reaches 98.9%.

arXiv comment11 pages, 4 figures

References in corpus (6)

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Exploiting Similarities among Languages for Machine Translation
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Automatic Identification of Document Translations in Large Multilingual Document Collections

Cited by in corpus (13)

• XNLI: Evaluating Cross-lingual Sentence Representations
• Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation
• Filtering and Mining Parallel Data in a Joint Multilingual Space
• Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron
• EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning
• Simple and Effective Paraphrastic Similarity from Parallel Translations
• Towards Interlingua Neural Machine Translation
• XCMRC: Evaluating Cross-lingual Machine Reading Comprehension
• Bilingual Lexicon Induction via Unsupervised Bitext Construction and Word Alignment
• Lightweight Cross-Lingual Sentence Representation Learning