Memory-enhanced Decoder for Neural Machine Translation

Abstract

We propose to enhance the RNN decoder in a neural machine translator (NMT) with external memory, as a natural but powerful extension to the state in the decoding RNN. This memory-enhanced RNN decoder is called \textsc{MemDec}. At each time during decoding, \textsc{MemDec} will read from this memory and write to this memory once, both with content-based addressing. Unlike the unbounded memory in previous work\cite{RNNsearch} to store the representation of source sentence, the memory in \textsc{MemDec} is a matrix with pre-determined size designed to better capture the information important for the decoding process at each time step. Our empirical study on Chinese-English translation shows that it can improve by $4.8$ BLEU upon Groundhog and $5.3$ BLEU upon on Moses, yielding the best performance achieved with the same training set.

11 pages

References in corpus (8)

• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• How to Construct Deep Recurrent Neural Networks
• Long Short-Term Memory-Networks for Machine Reading
• Modeling Coverage for Neural Machine Translation
• A Deep Memory-based Architecture for Sequence-to-Sequence Learning
• Context-Dependent Translation Selection Using Convolutional Neural Network

Cited by in corpus (11)

• Self-Attentive Residual Decoder for Neural Machine Translation
• Memory-augmented Neural Machine Translation
• Deconvolution-Based Global Decoding for Neural Machine Translation
• Neural Machine Translation with Key-Value Memory-Augmented Attention
• Neural Entity Reasoner for Global Consistency in NER
• Closed-Book Training to Improve Summarization Encoder Memory
• Learning to Discriminate Noises for Incorporating External Information in Neural Machine Translation
• Fusing Recency into Neural Machine Translation with an Inter-Sentence Gate Model
• Memory-Augmented Neural Networks for Machine Translation
• Modeling Past and Future for Neural Machine Translation
• Learning When to Concentrate or Divert Attention: Self-Adaptive Attention Temperature for Neural Machine Translation