Meta-Learning for Low-Resource Neural Machine Translation

Abstract

In this paper, we propose to extend the recently introduced model-agnostic meta-learning algorithm (MAML) for low-resource neural machine translation (NMT). We frame low-resource translation as a meta-learning problem, and we learn to adapt to low-resource languages based on multilingual high-resource language tasks. We use the universal lexical representation~\citep{gu2018universal} to overcome the input-output mismatch across different languages. We evaluate the proposed meta-learning strategy using eighteen European languages (Bg, Cs, Da, De, El, Es, Et, Fr, Hu, It, Lt, Nl, Pl, Pt, Sk, Sl, Sv and Ru) as source tasks and five diverse languages (Ro, Lv, Fi, Tr and Ko) as target tasks. We show that the proposed approach significantly outperforms the multilingual, transfer learning based approach~\citep{zoph2016transfer} and enables us to train a competitive NMT system with only a fraction of training examples. For instance, the proposed approach can achieve as high as 22.04 BLEU on Romanian-English WMT'16 by seeing only 16,000 translated words (~600 parallel sentences).

arXiv commentAccepted as a full paper at EMNLP 2018

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Convolutional Sequence to Sequence Learning
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Matching Networks for One Shot Learning
• A Simple Neural Attentive Meta-Learner
• Dual Learning for Machine Translation
• On Using Monolingual Corpora in Neural Machine Translation
• Enriching Word Vectors with Subword Information
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• Unsupervised Machine Translation Using Monolingual Corpora Only
• A Teacher-Student Framework for Zero-Resource Neural Machine Translation
• Unsupervised Neural Machine Translation
• Emergent Translation in Multi-Agent Communication
• Universal Neural Machine Translation for Extremely Low Resource Languages
• Neural Machine Translation with Pivot Languages
• Unsupervised Neural Machine Translation with Weight Sharing
• Zero-Resource Neural Machine Translation with Multi-Agent Communication Game

Cited by in corpus (17)

• Language Models are Few-Shot Learners
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• A Focus on Neural Machine Translation for African Languages
• Low-resource Languages: A Review of Past Work and Future Challenges
• Data Efficient Direct Speech-to-Text Translation with Modality Agnostic Meta-Learning
• Towards Neural Machine Translation for African Languages
• Improved Zero-shot Neural Machine Translation via Ignoring Spurious Correlations
• How Important is the Train-Validation Split in Meta-Learning?
• $k$Folden: $k$-Fold Ensemble for Out-Of-Distribution Detection
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• One Shot Learning for Speech Separation
• Meta-Learning with MAML on Trees
• A Survey on Low-Resource Neural Machine Translation
• Multi-accent Speech Separation with One Shot Learning
• Challenge Closed-book Science Exam: A Meta-learning Based Question Answering System
• Active Learning for Massively Parallel Translation of Constrained Text into Low Resource Languages
• A Dynamic Head Importance Computation Mechanism for Neural Machine Translation