Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information

Abstract

We investigate the following question for machine translation (MT): can we develop a single universal MT model to serve as the common seed and obtain derivative and improved models on arbitrary language pairs? We propose mRASP, an approach to pre-train a universal multilingual neural machine translation model. Our key idea in mRASP is its novel technique of random aligned substitution, which brings words and phrases with similar meanings across multiple languages closer in the representation space. We pre-train a mRASP model on 32 language pairs jointly with only public datasets. The model is then fine-tuned on downstream language pairs to obtain specialized MT models. We carry out extensive experiments on 42 translation directions across a diverse settings, including low, medium, rich resource, and as well as transferring to exotic language pairs. Experimental results demonstrate that mRASP achieves significant performance improvement compared to directly training on those target pairs. It is the first time to verify that multiple low-resource language pairs can be utilized to improve rich resource MT. Surprisingly, mRASP is even able to improve the translation quality on exotic languages that never occur in the pre-training corpus. Code, data, and pre-trained models are available at https://github.com/linzehui/mRASP.

arXiv commentEMNLP 2020

References in corpus (7)

• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Massively Multilingual Word Embeddings
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Multilingual Neural Machine Translation with Knowledge Distillation
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks

Cited by in corpus (19)

• XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
• Pretrained Language Models for Text Generation: A Survey
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Autocorrect in the Process of Translation -- Multi-task Learning Improves Dialogue Machine Translation
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Learning Language Specific Sub-network for Multilingual Machine Translation
• Cross-lingual Supervision Improves Unsupervised Neural Machine Translation
• LightSeq: A High Performance Inference Library for Transformers
• Improving Neural Machine Translation by Bidirectional Training
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• Cross-Lingual Language Model Meta-Pretraining
• Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation
• Counter-Interference Adapter for Multilingual Machine Translation
• Competence-based Curriculum Learning for Multilingual Machine Translation
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Multilingual Translation via Grafting Pre-trained Language Models
• A Survey on Green Deep Learning
• Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages