Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
arXiv:1906.01181
Abstract
Zero-shot translation, translating between language pairs on which a Neural Machine Translation (NMT) system has never been trained, is an emergent property when training the system in multilingual settings. However, naive training for zero-shot NMT easily fails, and is sensitive to hyper-parameter setting. The performance typically lags far behind the more conventional pivot-based approach which translates twice using a third language as a pivot. In this work, we address the degeneracy problem due to capturing spurious correlations by quantitatively analyzing the mutual information between language IDs of the source and decoded sentences. Inspired by this analysis, we propose to use two simple but effective approaches: (1) decoder pre-training; (2) back-translation. These methods show significant improvement (4~22 BLEU points) over the vanilla zero-shot translation on three challenging multilingual datasets, and achieve similar or better results than the pivot-based approach.
Accepted by ACL 2019
References in corpus (10)
- Sequence to Sequence Learning with Neural Networks
- Cross-lingual Language Model Pretraining
- A Teacher-Student Framework for Zero-Resource Neural Machine Translation
- The Missing Ingredient in Zero-Shot Neural Machine Translation
- Emergent Translation in Multi-Agent Communication
- Universal Neural Machine Translation for Extremely Low Resource Languages
- Neural Machine Translation with Pivot Languages
- Meta-Learning for Low-Resource Neural Machine Translation
- Zero-Shot Dual Machine Translation
- Zero-Resource Neural Machine Translation with Multi-Agent Communication Game
Cited by in corpus (7)
- Multilingual Denoising Pre-training for Neural Machine Translation
- Beyond English-Centric Multilingual Machine Translation
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
- Low Resource Neural Machine Translation: A Benchmark for Five African Languages
- Self-Learning for Zero Shot Neural Machine Translation
- Improving Multilingual Translation by Representation and Gradient Regularization