Decoding-History-Based Adaptive Control of Attention for Neural Machine Translation

Abstract

Attention-based sequence-to-sequence model has proved successful in Neural Machine Translation (NMT). However, the attention without consideration of decoding history, which includes the past information in the decoder and the attention mechanism, often causes much repetition. To address this problem, we propose the decoding-history-based Adaptive Control of Attention (ACA) for the NMT model. ACA learns to control the attention by keeping track of the decoding history and the current information with a memory vector, so that the model can take the translated contents and the current information into consideration. Experiments on Chinese-English translation and the English-Vietnamese translation have demonstrated that our model significantly outperforms the strong baselines. The analysis shows that our model is capable of generating translation with less repetition and higher accuracy. The code will be available at https://github.com/lancopku

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Online and Linear-Time Attention by Enforcing Monotonic Alignments

Cited by in corpus (8)

• Attention in Natural Language Processing
• Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
• Global Encoding for Abstractive Summarization
• Deconvolution-Based Global Decoding for Neural Machine Translation
• Bag-of-Words as Target for Neural Machine Translation
• Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
• Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation
• Future-Prediction-Based Model for Neural Machine Translation