Learning Language Specific Sub-network for Multilingual Machine Translation

Abstract

Multilingual neural machine translation aims at learning a single translation model for multiple languages. These jointly trained models often suffer from performance degradation on rich-resource language pairs. We attribute this degeneration to parameter interference. In this paper, we propose LaSS to jointly train a single unified multilingual MT model. LaSS learns Language Specific Sub-network (LaSS) for each language pair to counter parameter interference. Comprehensive experiments on IWSLT and WMT datasets with various Transformer architectures show that LaSS obtains gains on 36 language pairs by up to 1.2 BLEU. Besides, LaSS shows its strong generalization performance at easy extension to new language pairs and zero-shot translation.LaSS boosts zero-shot translation with an average of 8.3 BLEU on 30 language pairs. Codes and trained models are available at https://github.com/NLP-Playground/LaSS.

arXiv commentACL2021

References in corpus (13)

• Overcoming catastrophic forgetting in neural networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Rethinking the Value of Network Pruning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Multilingual Denoising Pre-training for Neural Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Learning multiple visual domains with residual adapters
• Beyond English-Centric Multilingual Machine Translation
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Information
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Deep Transformers with Latent Depth