Towards Interlingua Neural Machine Translation
arXiv:1905.06831
Abstract
Common intermediate language representation in neural machine translation can be used to extend bilingual to multilingual systems by incremental training. In this paper, we propose a new architecture based on introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we are able to train multiple encoders and decoders for each language, sharing a common intermediate representation. Translation results on the low-resourced tasks (Turkish-English and Kazakh-English tasks, from the popular Workshop on Machine Translation benchmark) show the following BLEU improvements up to 2.8. However, results on a larger dataset (Russian-English and Kazakh-English, from the same baselines) show BLEU loses if the same amount. While our system is only providing improvements for the low-resourced tasks in terms of translation quality, our system is capable of quickly deploying new language pairs without retraining the rest of the system, which may be a game-changer in some situations (i.e. in a disaster crisis where international help is required towards a small region or to develop some translation system for a client). Precisely, what is most relevant from our architecture is that it is capable of: (1) reducing the number of production systems, with respect to the number of languages, from quadratic to linear (2) incrementally adding a new language in the system without retraining languages previously there and (3) allowing for translations from the new language to all the others present in the system
arXiv commentarXiv admin note: substantial text overlap with arXiv:1810.06351
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Neural Discrete Representation Learning
- Unsupervised Statistical Machine Translation
- Phrase-Based & Neural Unsupervised Machine Translation
- Unsupervised Machine Translation Using Monolingual Corpora Only
- Fast Decoding in Sequence Models using Discrete Latent Variables
- XNLI: Evaluating Cross-lingual Sentence Representations
- Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
- Unsupervised Neural Machine Translation
- The Missing Ingredient in Zero-Shot Neural Machine Translation
- Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
- An Empirical Analysis of NMT-Derived Interlingual Embeddings and their Use in Parallel Sentence Identification
- Variational Neural Machine Translation
- A neural interlingua for multilingual machine translation