Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model
arXiv:1601.03317
Abstract
Neural machine translation has shown very promising results lately. Most NMT models follow the encoder-decoder framework. To make encoder-decoder models more flexible, attention mechanism was introduced to machine translation and also other tasks like speech recognition and image captioning. We observe that the quality of translation by attention-based encoder-decoder can be significantly damaged when the alignment is incorrect. We attribute these problems to the lack of distortion and fertility models. Aiming to resolve these problems, we propose new variations of attention-based encoder-decoder and compare them with other models on machine translation. Our proposed method achieved an improvement of 2 BLEU points over the original attention-based encoder-decoder.
11 pages, updated details
References in corpus (8)
- Sequence to Sequence Learning with Neural Networks
- On the difficulty of training Recurrent Neural Networks
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Sequence Transduction with Recurrent Neural Networks
- Recurrent Models of Visual Attention
- End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Cited by in corpus (12)
- Modeling Coverage for Neural Machine Translation
- Temporal Attention Model for Neural Machine Translation
- Bridging Neural Machine Translation and Bilingual Dictionaries
- Learning for Biomedical Information Extraction: Methodological Review of Recent Advances
- Neural Machine Translation with Supervised Attention
- Interactive Attention for Neural Machine Translation
- Neural Machine Translation with Recurrent Attention Modeling
- Neural Machine Translation with Key-Value Memory-Augmented Attention
- Neural Machine Translation Advised by Statistical Machine Translation
- Controlling Utterance Length in NMT-based Word Segmentation with Attention
- Neural Machine Translation: A Review and Survey
- Phrase Table as Recommendation Memory for Neural Machine Translation