Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

Abstract

Transfer learning or multilingual model is essential for low-resource neural machine translation (NMT), but the applicability is limited to cognate languages by sharing their vocabularies. This paper shows effective techniques to transfer a pre-trained NMT model to a new, unrelated language without shared vocabularies. We relieve the vocabulary mismatch by using cross-lingual word embedding, train a more language-agnostic encoder by injecting artificial noises, and generate synthetic data easily from the pre-training data without back-translation. Our methods do not require restructuring the vocabulary or retraining the model. We improve plain NMT transfer by up to +5.1% BLEU in five low-resource translation tasks, outperforming multilingual joint training by a large margin. We also provide extensive ablation studies on pre-trained embedding, synthetic data, vocabulary size, and parameter freezing for a better understanding of NMT transfer.

arXiv commentACL 2019 camera-ready

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Cross-lingual Language Model Pretraining
• Theoretical Models of Learning to Learn
• Unsupervised Statistical Machine Translation
• Sockeye: A Toolkit for Neural Machine Translation
• Fast Domain Adaptation for Neural Machine Translation
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation

Cited by in corpus (3)

• Self-Attentive Constituency Parsing for UCCA-based Semantic Parsing
• A Survey on Low-Resource Neural Machine Translation
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model