Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation

Abstract

We present a simple method to improve neural translation of a low-resource language pair using parallel data from a related, also low-resource, language pair. The method is based on the transfer method of Zoph et al., but whereas their method ignores any source vocabulary overlap, ours exploits it. First, we split words using Byte Pair Encoding (BPE) to increase vocabulary overlap. Then, we train a model on the first language pair and transfer its parameters, including its source word embeddings, to another model and continue training on the second language pair. Our experiments show that transfer learning helps word-based translation only slightly, but when used on top of a much stronger BPE baseline, it yields larger improvements of up to 4.3 BLEU.

References in corpus (1)

• ADADELTA: An Adaptive Learning Rate Method

Cited by in corpus (31)

• IndicBART: A Pre-trained Model for Indic Natural Language Generation
• Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation
• Multilingual Neural Machine Translation With Soft Decoupled Encoding
• Low-resource Languages: A Review of Past Work and Future Challenges
• WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models
• Low Resource Neural Machine Translation: A Benchmark for Five African Languages
• A Universal Parent Model for Low-Resource Neural Machine Translation Transfer
• Rapid Adaptation of Neural Machine Translation to New Languages
• Balancing Training for Multilingual Neural Machine Translation
• Low-Resource Neural Machine Translation for Southern African Languages
• Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary
• Investigating Multilingual NMT Representations at Scale
• Adapting Multilingual Neural Machine Translation to Unseen Languages
• BERT in Plutarch's Shadows
• Tag-less Back-Translation
• Target Conditioned Sampling: Optimizing Data Selection for Multilingual Neural Machine Translation
• Translating the Unseen? Yoruba-English MT in Low-Resource, Morphologically-Unmarked Settings
• A Brief Survey of Multilingual Neural Machine Translation
• A Survey on Low-Resource Neural Machine Translation
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Contextual Text Embeddings for Twi
• Neural Machine Translation: A Review and Survey
• Semi-supervised transfer learning for language expansion of end-to-end speech recognition models to low-resource languages
• Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages
• Revisiting Modularized Multilingual NMT to Meet Industrial Demands
• Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation
• Self-Learning for Zero Shot Neural Machine Translation
• Cross-lingual Word Embeddings beyond Zero-shot Machine Translation
• Unsupervised Spoken Term Discovery on Untranscribed Speech
• Predicting the Performance of Multilingual NLP Models
• "Wikily" Supervised Neural Translation Tailored to Cross-Lingual Tasks