Training Deeper Neural Machine Translation Models with Transparent Attention

Abstract

While current state-of-the-art NMT models, such as RNN seq2seq and Transformers, possess a large number of parameters, they are still shallow in comparison to convolutional models used for both text and vision applications. In this work we attempt to train significantly (2-3x) deeper Transformer and Bi-RNN encoders for machine translation. We propose a simple modification to the attention mechanism that eases the optimization of deeper models, and results in consistent gains of 0.7-1.1 BLEU on the benchmark WMT'14 English-German and WMT'15 Czech-English tasks for both architectures.

arXiv commentTo appear in EMNLP 2018

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Deep Residual Learning for Image Recognition
• Convolutional Sequence to Sequence Learning
• Densely Connected Convolutional Networks
• Highway Networks
• Deep Networks with Stochastic Depth
• The Power of Depth for Feedforward Neural Networks
• Identity Matters in Deep Learning
• Deep Neural Machine Translation with Linear Associative Unit

Cited by in corpus (11)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Deep Modular Co-Attention Networks for Visual Question Answering
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Robust Neural Machine Translation with Doubly Adversarial Inputs
• Deep Transformers with Latent Depth
• Simple, Scalable Adaptation for Neural Machine Translation
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• An Efficient Transformer Decoder with Compressed Sub-layers
• Marathi To English Neural Machine Translation With Near Perfect Corpus And Transformers
• OmniNet: Omnidirectional Representations from Transformers
• Character-based NMT with Transformer