A Comparison of Transformer and Recurrent Neural Networks on Multilingual Neural Machine Translation

Abstract

Recently, neural machine translation (NMT) has been extended to multilinguality, that is to handle more than one translation direction with a single system. Multilingual NMT showed competitive performance against pure bilingual systems. Notably, in low-resource settings, it proved to work effectively and efficiently, thanks to shared representation space that is forced across languages and induces a sort of transfer-learning. Furthermore, multilingual NMT enables so-called zero-shot inference across language pairs never seen at training time. Despite the increasing interest in this framework, an in-depth analysis of what a multilingual NMT model is capable of and what it is not is still missing. Motivated by this, our work (i) provides a quantitative and comparative analysis of the translations produced by bilingual, multilingual and zero-shot systems; (ii) investigates the translation quality of two of the currently dominant neural architectures in MT, which are the Recurrent and the Transformer ones; and (iii) quantitatively explores how the closeness between languages influences the zero-shot translation. Our analysis leverages multiple professional post-edits of automatic translations by several different systems and focuses both on automatic standard metrics (BLEU and TER) and on widely used error categories, which are lexical, morphology, and word order errors.

arXiv comment12 pages, to appear on the 27th International Conference on Computational Linguistics (COLING 2018)

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

Cited by in corpus (10)

• A Comparative Study on Transformer vs RNN in Speech Applications
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Learning from flowsheets: A generative transformer model for autocompletion of flowsheets
• Transformer based Grapheme-to-Phoneme Conversion
• Context in Neural Machine Translation: A Review of Models and Evaluations
• Distinguishing a planetary transit from false positives: a Transformer-based classification for planetary transit signals
• Exoplanet Transit Candidate Identification in TESS Full-Frame Images via a Transformer-Based Algorithm
• A Brief Survey of Multilingual Neural Machine Translation
• Fine-grained Human Evaluation of Transformer and Recurrent Approaches to Neural Machine Translation for English-to-Chinese
• Neural Machine Translation: A Review and Survey