Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation

Abstract

Over the last few years two promising research directions in low-resource neural machine translation (NMT) have emerged. The first focuses on utilizing high-resource languages to improve the quality of low-resource languages via multilingual NMT. The second direction employs monolingual data with self-supervision to pre-train translation models, followed by fine-tuning on small amounts of supervised data. In this work, we join these two lines of research and demonstrate the efficacy of monolingual data with self-supervision in multilingual NMT. We offer three major results: (i) Using monolingual data significantly boosts the translation quality of low-resource languages in multilingual models. (ii) Self-supervision improves zero-shot translation quality in multilingual models. (iii) Leveraging monolingual data with self-supervision provides a viable path towards adding new languages to multilingual models, getting up to 33 BLEU on ro-en translation without any parallel data or back-translation.

References in corpus (7)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Semi-supervised Sequence Learning
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• The Missing Ingredient in Zero-Shot Neural Machine Translation

Cited by in corpus (17)

• Beyond English-Centric Multilingual Machine Translation
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• MURAL: Multimodal, Multitask Retrieval Across Languages
• Improving Multilingual Neural Machine Translation For Low-Resource Languages: French,English - Vietnamese
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
• Learning Language Specific Sub-network for Multilingual Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Rapid Domain Adaptation for Machine Translation with Monolingual Data
• Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• Towards Universality in Multilingual Text Rewriting
• Token-wise Curriculum Learning for Neural Machine Translation
• Unsupervised Paraphrasing with Pretrained Language Models
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
• Self-Learning for Zero Shot Neural Machine Translation
• Counter-Interference Adapter for Multilingual Machine Translation