More Parameters? No Thanks!
arXiv:2107.09622
Abstract
This work studies the long-standing problems of model capacity and negative interference in multilingual neural machine translation MNMT. We use network pruning techniques and observe that pruning 50-70% of the parameters from a trained MNMT model results only in a 0.29-1.98 drop in the BLEU score. Suggesting that there exist large redundancies even in MNMT models. These observations motivate us to use the redundant parameters and counter the interference problem efficiently. We propose a novel adaptation strategy, where we iteratively prune and retrain the redundant parameters of an MNMT to improve bilingual representations while retaining the multilinguality. Negative interference severely affects high resource languages, and our method alleviates it without any additional adapter modules. Hence, we call it parameter-free adaptation strategy, paving way for the efficient adaptation of MNMT. We demonstrate the effectiveness of our method on a 9 language MNMT trained on TED talks, and report an average improvement of +1.36 BLEU on high resource pairs. Code will be released here.
References in corpus (7)
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Overcoming Catastrophic Forgetting by Incremental Moment Matching
- Multilingual Neural Machine Translation with Knowledge Distillation
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
- Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation
- Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
- Exploring Pair-Wise NMT for Indian Languages