Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Abstract

Recent work demonstrates the potential of multilingual pretraining of creating one model that can be used for various tasks in different languages. Previous work in multilingual pretraining has demonstrated that machine translation systems can be created by finetuning on bitext. In this work, we show that multilingual translation models can be created through multilingual finetuning. Instead of finetuning on one direction, a pretrained model is finetuned on many directions at the same time. Compared to multilingual models trained from scratch, starting from pretrained models incorporates the benefits of large quantities of unlabeled monolingual data, which is particularly important for low resource languages where bitext is not available. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance. We double the number of languages in mBART to support multilingual machine translation models of 50 languages. Finally, we create the ML50 benchmark, covering low, mid, and high resource languages, to facilitate reproducible research by standardizing training and evaluation data. On ML50, we demonstrate that multilingual finetuning improves on average 1 BLEU over the strongest baselines (being either multilingual from scratch or bilingual finetuning) while improving 9.3 BLEU on average over bilingual baselines from scratch.

10 pages (main) + 5 pages (appendices). 9 tables and 2 figures

References in corpus (9)

• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• XNLI: Evaluating Cross-lingual Sentence Representations
• The Missing Ingredient in Zero-Shot Neural Machine Translation
• Cross-lingual Retrieval for Iterative Self-Supervised Training
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations

Cited by in corpus (26)

• DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
• Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark
• AVATAR: A Parallel Corpus for Java-Python Program Translation
• Modelling Latent Translations for Cross-Lingual Transfer
• Are the Multilingual Models Better? Improving Czech Sentiment with Transformers
• The Effect of Domain and Diacritics in Yorùbá-English Neural Machine Translation
• Robust Optimization for Multilingual Translation with Imbalanced Data
• A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention
• Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
• A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction
• Lost In Translation: Generating Adversarial Examples Robust to Round-Trip Translation
• Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages
• On the Copying Behaviors of Pre-Training for Neural Machine Translation
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
• mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer
• Multilingual Translation via Grafting Pre-trained Language Models
• IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
• On the Complementarity between Pre-Training and Back-Translation for Neural Machine Translation
• Multilingual AMR Parsing with Noisy Knowledge Distillation
• An Empirical Investigation of Multi-bridge Multilingual NMT models
• Don't Go Far Off: An Empirical Study on Neural Poetry Translation
• Improving Multilingual Translation by Representation and Gradient Regularization
• Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation