Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Abstract

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

ACL2020

References in corpus (3)

• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• The Missing Ingredient in Zero-Shot Neural Machine Translation
• Multilingual NMT with a language-independent attention bridge

Cited by in corpus (32)

• Beyond English-Centric Multilingual Machine Translation
• Many-to-English Machine Translation Tools, Data, and Pretrained Models
• Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
• Improving Zero-Shot Translation by Disentangling Positional Information
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• Language Models are Good Translators
• End-to-end Speech Translation via Cross-modal Progressive Training
• Learning Shared Semantic Space for Speech-to-Text Translation
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation
• Back-translation for Large-Scale Multilingual Machine Translation
• Learning Language Specific Sub-network for Multilingual Machine Translation
• Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
• Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task
• Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
• Language Tags Matter for Zero-Shot Neural Machine Translation
• Extremely low-resource machine translation for closely related languages
• Towards Universality in Multilingual Text Rewriting
• Simultaneous Multi-Pivot Neural Machine Translation
• nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
• More Parameters? No Thanks!
• Competence-based Curriculum Learning for Multilingual Machine Translation
• HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints
• Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables
• Counter-Interference Adapter for Multilingual Machine Translation
• Multilingual Translation via Grafting Pre-trained Language Models
• The ELITR ECA Corpus
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference
• Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits
• From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text