Simple, Scalable Adaptation for Neural Machine Translation

Abstract

Fine-tuning pre-trained Neural Machine Translation (NMT) models is the dominant approach for adapting to new languages and domains. However, fine-tuning requires adapting and maintaining a separate model for each target task. We propose a simple yet efficient approach for adaptation in NMT. Our proposed approach consists of injecting tiny task specific adapter layers into a pre-trained model. These lightweight adapters, with just a small fraction of the original model size, adapt the model to multiple individual tasks simultaneously. We evaluate our approach on two tasks: (i) Domain Adaptation and (ii) Massively Multilingual NMT. Experiments on domain adaptation demonstrate that our proposed approach is on par with full fine-tuning on various domains, dataset sizes and model capacities. On a massively multilingual dataset of 103 languages, our adaptation approach bridges the gap between individual bilingual models and one massively multilingual model for most language pairs, paving the way towards universal machine translation.

arXiv commentEMNLP 2019

References in corpus (12)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• Learning multiple visual domains with residual adapters
• A Survey of Domain Adaptation for Neural Machine Translation
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Fast Domain Adaptation for Neural Machine Translation
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Parameter-Efficient Transfer Learning for NLP
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation
• Training Deeper Neural Machine Translation Models with Transparent Attention

Cited by in corpus (9)

• BASE Layers: Simplifying Training of Large, Sparse Models
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• CoCon: A Self-Supervised Approach for Controlled Text Generation
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Training Neural Networks with Fixed Sparse Masks
• A Survey on Low-Resource Neural Machine Translation
• Meta-Learning for Few-Shot NMT Adaptation
• Revisiting Modularized Multilingual NMT to Meet Industrial Demands