Modeling Source Syntax for Neural Machine Translation

Abstract

Even though a linguistics-free sequence to sequence model in neural machine translation (NMT) has certain capability of implicitly learning syntactic information of source sentences, this paper shows that source syntax can be explicitly incorporated into NMT effectively to provide further improvements. Specifically, we linearize parse trees of source sentences to obtain structural label sequences. On the basis, we propose three different sorts of encoders to incorporate source syntax into NMT: 1) Parallel RNN encoder that learns word and label annotation vectors parallelly; 2) Hierarchical RNN encoder that learns word and label annotation vectors in a two-level hierarchy; and 3) Mixed RNN encoder that stitchingly learns word and label annotation vectors over sequences where words and labels are mixed. Experimentation on Chinese-to-English translation demonstrates that all the three proposed syntactic encoders are able to improve translation accuracy. It is interesting to note that the simplest RNN encoder, i.e., Mixed RNN encoder yields the best performance with an significant improvement of 1.4 BLEU points. Moreover, an in-depth analysis from several perspectives is provided to reveal how source syntax benefits NMT.

Accepted by ACL 2017

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ADADELTA: An Adaptive Learning Rate Method
• Modeling Coverage for Neural Machine Translation
• Neural Machine Translation with Reconstruction
• Tree-to-Sequence Attentional Neural Machine Translation
• Supervised Attentions for Neural Machine Translation

Cited by in corpus (14)

• Learning to Remember Translation History with a Continuous Cache
• Variational Recurrent Neural Machine Translation
• Enhancing Machine Translation with Dependency-Aware Self-Attention
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Recurrent Graph Syntax Encoder for Neural Machine Translation
• Improving AMR Parsing with Sequence-to-Sequence Pre-training
• Refining Source Representations with Relation Networks for Neural Machine Translation
• Neural Machine Translation with Source-Side Latent Graph Parsing
• Multi-Source Syntactic Neural Machine Translation
• Evaluating the Utility of Hand-crafted Features in Sequence Labelling
• Inducing Grammars with and for Neural Machine Translation
• Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings
• Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations
• On Tree-Based Neural Sentence Modeling