Improving Multilingual Translation by Representation and Gradient Regularization

Abstract

Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations -- commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available.

EMNLP 2021 (Oral). Code and data: https://github.com/yilinyang7/fairseq_multi_fix

References in corpus (11)

• An Overview of Multi-Task Learning in Deep Neural Networks
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Gradient-guided Loss Masking for Neural Machine Translation
• Multi-task Learning for Multilingual Neural Machine Translation
• On the Sub-Layer Functionalities of Transformer Decoder