Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation

Abstract

The data scarcity in low-resource languages has become a bottleneck to building robust neural machine translation systems. Fine-tuning a multilingual pre-trained model (e.g., mBART (Liu et al., 2020)) on the translation task is a good approach for low-resource languages; however, its performance will be greatly limited when there are unseen languages in the translation pairs. In this paper, we present a continual pre-training (CPT) framework on mBART to effectively adapt it to unseen languages. We first construct noisy mixed-language text from the monolingual corpus of the target language in the translation pair to cover both the source and target languages, and then, we continue pre-training mBART to reconstruct the original monolingual text. Results show that our method can consistently improve the fine-tuning performance upon the mBART baseline, as well as other strong baselines, across all tested low-resource translation pairs containing unseen languages. Furthermore, our approach also boosts the performance on translation pairs where both languages are seen in the original mBART's pre-training. The code is available at https://github.com/zliucr/cpt-nmt.

arXiv commentAccepted in Findings of ACL 2021

References in corpus (16)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• mT5: A massively multilingual pre-trained text-to-text transformer
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Beyond English-Centric Multilingual Machine Translation
• A Closer Look at Few-shot Classification
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Neural machine translation for low-resource languages
• CrossNER: Evaluating Cross-Domain Named Entity Recognition
• Exploiting Out-of-Domain Parallel Data through Multilingual Transfer Learning for Low-Resource Neural Machine Translation
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling
• AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization