Deep Neural Machine Translation with Linear Associative Unit

Abstract

Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling complex functions and capturing complex linguistic structures. However NMT systems with deep architecture in their encoder or decoder RNNs often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often make the optimization much more difficult. To address this problem we propose novel linear associative units (LAU) to reduce the gradient propagation length inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs utilizes linear associative connections between input and output of the recurrent unit, which allows unimpeded information flow through both space and time direction. The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported results in the same setting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art.

10 pages, ACL 2017

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ADADELTA: An Adaptive Learning Rate Method
• Training Very Deep Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Grid Long Short-Term Memory
• How to Construct Deep Recurrent Neural Networks
• Addressing the Rare Word Problem in Neural Machine Translation
• Minimum Risk Training for Neural Machine Translation
• Variational Neural Machine Translation
• Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation

Cited by in corpus (6)

• A Survey of Deep Learning Techniques for Neural Machine Translation
• Asynchronous Bidirectional Decoding for Neural Machine Translation
• Towards Robust Neural Machine Translation
• Training Deeper Neural Machine Translation Models with Transparent Attention
• Unsupervised Neural Machine Translation with Weight Sharing
• Variational Recurrent Neural Machine Translation