Temporal Attention Model for Neural Machine Translation
arXiv:1608.02927
Abstract
Attention-based Neural Machine Translation (NMT) models suffer from attention deficiency issues as has been observed in recent research. We propose a novel mechanism to address some of these limitations and improve the NMT attention. Specifically, our approach memorizes the alignments temporally (within each sentence) and modulates the attention with the accumulated temporal memory, as the decoder generates the candidate translation. We compare our approach against the baseline NMT model and two other related approaches that address this issue either explicitly or implicitly. Large-scale experiments on two language pairs show that our approach achieves better and robust gains over the baseline and related NMT approaches. Our model further outperforms strong SMT baselines in some settings even without using ensembles.
8 pages
References in corpus (5)
- Neural Machine Translation by Jointly Learning to Align and Translate
- Sequence to Sequence Learning with Neural Networks
- ADADELTA: An Adaptive Learning Rate Method
- End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
- Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model
Cited by in corpus (10)
- Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
- Weakly Labelled AudioSet Tagging with Attention Neural Networks
- Neural Abstractive Text Summarization with Sequence-to-Sequence Models
- Query-Based Abstractive Summarization Using Neural Networks
- Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information
- Multi-stream Network With Temporal Attention For Environmental Sound Classification
- Learning Frame Level Attention for Environmental Sound Classification
- Temporal Attention-Gated Model for Robust Sequence Classification
- Controlling Utterance Length in NMT-based Word Segmentation with Attention
- Generating Classical Chinese Poems from Vernacular Chinese