The Missing Ingredient in Zero-Shot Neural Machine Translation

Abstract

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Dual Learning for Machine Translation
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Effective Strategies in Zero-Shot Neural Machine Translation

Cited by in corpus (3)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron