Refining Source Representations with Relation Networks for Neural Machine Translation

Abstract

Although neural machine translation with the encoder-decoder framework has achieved great success recently, it still suffers drawbacks of forgetting distant information, which is an inherent disadvantage of recurrent neural network structure, and disregarding relationship between source words during encoding step. Whereas in practice, the former information and relationship are often useful in current step. We target on solving these problems and thus introduce relation networks to learn better representations of the source. The relation networks are able to facilitate memorization capability of recurrent neural network via associating source words with each other, this would also help retain their relationships. Then the source representations and all the relations are fed into the attention component together while decoding, with the main encoder-decoder framework unchanged. Experiments on several datasets show that our method can improve the translation performance significantly over the conventional encoder-decoder model and even outperform the approach involving supervised syntactic knowledge.

arXiv comment12pages, 7 figures, accepted for COLING-2018. arXiv admin note: substantial text overlap with arXiv:1709.03980

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• ADADELTA: An Adaptive Learning Rate Method
• Convolutional Sequence to Sequence Learning
• A simple neural network module for relational reasoning
• Syntactically Guided Neural Machine Translation
• A Convolutional Encoder Model for Neural Machine Translation
• Modeling Source Syntax for Neural Machine Translation
• Variational Recurrent Neural Machine Translation