Universal Neural Machine Translation for Extremely Low Resource Languages

Abstract

In this paper, we propose a new universal machine translation approach focusing on languages with a limited amount of parallel data. Our proposed approach utilizes a transfer-learning approach to share lexical and sentence level representations across multiple source languages into one target language. The lexical part is shared through a Universal Lexical Representation to support multilingual word-level sharing. The sentence-level sharing is represented by a model of experts from all source languages that share the source encoders with all other languages. This enables the low-resource language to utilize the lexical and sentence representations of the higher resource languages. Our approach is able to achieve 23 BLEU on Romanian-English WMT2016 using a tiny parallel corpus of 6k sentences, compared to the 18 BLEU of strong baseline system which uses multilingual training and back-translation. Furthermore, we show that the proposed approach can achieve almost 20 BLEU on the same dataset through fine-tuning a pre-trained multi-lingual system in a zero-shot setting.

arXiv commentNAACL-HLT 2018

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Character-Aware Neural Language Models
• Achieving Human Parity on Automatic Chinese to English News Translation
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation

Cited by in corpus (23)

• Multilingual Denoising Pre-training for Neural Machine Translation
• Phrase-Based & Neural Unsupervised Machine Translation
• Understanding Back-Translation at Scale
• XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering
• Low-resource Languages: A Review of Past Work and Future Challenges
• Meta-Learning for Low-Resource Neural Machine Translation
• From English To Foreign Languages: Transferring Pre-trained Language Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• Towards Neural Machine Translation for African Languages
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• Rapid Adaptation of Neural Machine Translation to New Languages
• Fundamental Limits and Tradeoffs in Invariant Representation Learning
• Transfer Learning in Multilingual Neural Machine Translation with Dynamic Vocabulary
• Low-Resource Neural Machine Translation for Southern African Languages
• Language Graph Distillation for Low-Resource Machine Translation
• Towards Fair Federated Learning with Zero-Shot Data Augmentation
• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• Chinese-Portuguese Machine Translation: A Study on Building Parallel Corpora from Comparable Texts
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
• A Survey on Low-Resource Neural Machine Translation
• Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation
• Density Matching for Bilingual Word Embedding
• Language-Independent Representor for Neural Machine Translation