Improving Multilingual Translation by Representation and Gradient Regularization

Abstract

Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations -- commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available.

arXiv commentEMNLP 2021 (Oral). Code and data: https://github.com/yilinyang7/fairseq_multi_fix

References in corpus (20)

• Adam: A Method for Stochastic Optimization
• An Overview of Multi-Task Learning in Deep Neural Networks
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Gradient Surgery for Multi-Task Learning
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• Multilingual Neural Machine Translation with Task-Specific Attention
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Learning to Parse and Translate Improves Neural Machine Translation
• Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning
• Breaking the Beam Search Curse: A Study of (Re-)Scoring Methods and Stopping Criteria for Neural Machine Translation
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Contextual Parameter Generation for Universal Neural Machine Translation
• Balancing Training for Multilingual Neural Machine Translation
• Improving Zero-Shot Translation by Disentangling Positional Information
• Gradient-guided Loss Masking for Neural Machine Translation
• Multi-task Learning for Multilingual Neural Machine Translation
• On the Sub-Layer Functionalities of Transformer Decoder