Look-ahead Attention for Generation in Neural Machine Translation

Abstract

The attention model has become a standard component in neural machine translation (NMT) and it guides translation process by selectively focusing on parts of the source sentence when predicting each target word. However, we find that the generation of a target word does not only depend on the source sentence, but also rely heavily on the previous generated target words, especially the distant words which are difficult to model by using recurrent neural networks. To solve this problem, we propose in this paper a novel look-ahead attention mechanism for generation in NMT, which aims at directly capturing the dependency relationship between target words. We further design three patterns to integrate our look-ahead attention into the conventional attention model. Experiments on NIST Chinese-to-English and WMT English-to-German translation tasks show that our proposed look-ahead attention mechanism achieves substantial improvements over state-of-the-art baselines.

arXiv comment12 pages, 5 figures

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• A Deep Reinforced Model for Abstractive Summarization
• Long Short-Term Memory-Networks for Machine Reading
• Modeling Coverage for Neural Machine Translation
• Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions
• Supervised Attentions for Neural Machine Translation
• Context Gates for Neural Machine Translation