Explicit Cross-lingual Pre-training for Unsupervised Machine Translation

Abstract

Pre-training has proven to be effective in unsupervised machine translation due to its ability to model deep context information in cross-lingual scenarios. However, the cross-lingual information obtained from shared BPE spaces is inexplicit and limited. In this paper, we propose a novel cross-lingual pre-training method for unsupervised machine translation by incorporating explicit cross-lingual training signals. Specifically, we first calculate cross-lingual n-gram embeddings and infer an n-gram translation table from them. With those n-gram translation pairs, we propose a new pre-training model called Cross-lingual Masked Language Model (CMLM), which randomly chooses source n-grams in the input text stream and predicts their translation candidates at each time step. Experiments show that our method can incorporate beneficial cross-lingual information into pre-trained models. Taking pre-trained CMLM models as the encoder and decoder, we significantly improve the performance of unsupervised machine translation.

arXiv commentAccepted to EMNLP2019; 10 pages, 2 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Cross-lingual Language Model Pretraining
• A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Unsupervised Statistical Machine Translation
• Word Translation Without Parallel Data
• Unsupervised Machine Translation Using Monolingual Corpora Only
• XNLI: Evaluating Cross-lingual Sentence Representations
• Unsupervised Neural Machine Translation
• Unsupervised Neural Machine Translation Initialized by Unsupervised Statistical Machine Translation
• Unsupervised Neural Machine Translation with SMT as Posterior Regularization