The Missing Ingredient in Zero-Shot Neural Machine Translation

Abstract

Multilingual Neural Machine Translation (NMT) models are capable of translating between multiple source and target languages. Despite various approaches to train such models, they have difficulty with zero-shot translation: translating between language pairs that were not together seen during training. In this paper we first diagnose why state-of-the-art multilingual NMT models that rely purely on parameter sharing, fail to generalize to unseen language pairs. We then propose auxiliary losses on the NMT encoder that impose representational invariance across languages. Our simple approach vastly improves zero-shot translation quality without regressing on supervised directions. For the first time, on WMT14 English-FrenchGerman, we achieve zero-shot performance that is on par with pivoting. We also demonstrate the easy scalability of our approach to multiple languages on the IWSLT 2017 shared task.

References in corpus (14)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Convolutional Sequence to Sequence Learning
• Dual Learning for Machine Translation
• BilBOWA: Fast Bilingual Distributed Representations without Word Alignments
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Multilingual Neural Machine Translation with Task-Specific Attention
• Effective Strategies in Zero-Shot Neural Machine Translation
• A neural interlingua for multilingual machine translation
• Zero-Shot Dual Machine Translation
• Triangular Architecture for Rare Language Translation

Cited by in corpus (26)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Zero-Shot Paraphrase Generation with Multilingual Language Models
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Improving Zero-Shot Translation by Disentangling Positional Information
• Bilingual Dictionary Based Neural Machine Translation without Using Parallel Sentences
• Subword Segmentation and a Single Bridge Language Affect Zero-Shot Neural Machine Translation
• Reference Language based Unsupervised Neural Machine Translation
• A Brief Survey of Multilingual Neural Machine Translation
• Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron
• A Survey on Low-Resource Neural Machine Translation
• Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
• Towards Interlingua Neural Machine Translation
• Language Tags Matter for Zero-Shot Neural Machine Translation
• Adapting High-resource NMT Models to Translate Low-resource Related Languages without Parallel Data
• Revisiting Modularized Multilingual NMT to Meet Industrial Demands
• Rethinking Zero-shot Neural Machine Translation: From a Perspective of Latent Variables
• An Empirical Investigation of Multi-bridge Multilingual NMT models
• Controlling Neural Machine Translation Formality with Synthetic Supervision
• Zero-Shot Language Transfer vs Iterative Back Translation for Unsupervised Machine Translation
• Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders
• Self-Learning for Zero Shot Neural Machine Translation