Contextual Parameter Generation for Universal Neural Machine Translation
arXiv:1808.08493
Abstract
We propose a simple modification to existing neural machine translation (NMT) models that enables using a single universal model to translate between multiple languages while allowing for language specific parameterization, and that can also be used for domain adaptation. Our approach requires no changes to the model architecture of a standard NMT system, but instead introduces a new component, the contextual parameter generator (CPG), that generates the parameters of the system (e.g., weights in a neural network). This parameter generator accepts source and target language embeddings as input, and generates the parameters for the encoder and the decoder, respectively. The rest of the model remains unchanged and is shared across all languages. We show how this simple modification enables the system to use monolingual data for training and also perform zero-shot translation. We further show it is able to surpass state-of-the-art performance for both the IWSLT-15 and IWSLT-17 datasets and that the learned language embeddings are able to uncover interesting relationships between languages.
arXiv commentPublished in the proceedings of Empirical Methods in Natural Language Processing (EMNLP), 2018
References in corpus (5)
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- A Teacher-Student Framework for Zero-Resource Neural Machine Translation
- Improving Zero-Shot Translation of Low-Resource Languages
- Zero-Resource Translation with Multi-Lingual Neural Machine Translation
Cited by in corpus (9)
- Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
- Multilingual NMT with a language-independent attention bridge
- A Study of Multilingual Neural Machine Translation
- A Survey on Low-Resource Neural Machine Translation
- WeightNet: Revisiting the Design Space of Weight Networks
- Zero-shot task adaptation by homoiconic meta-mapping
- Revisiting Modularized Multilingual NMT to Meet Industrial Demands
- Improving Multilingual Translation by Representation and Gradient Regularization
- Controlling Neural Machine Translation Formality with Synthetic Supervision