Regularizing Neural Machine Translation by Target-bidirectional Agreement

Abstract

Although Neural Machine Translation (NMT) has achieved remarkable progress in the past several years, most NMT systems still suffer from a fundamental shortcoming as in other sequence generation tasks: errors made early in generation process are fed as inputs to the model and can be quickly amplified, harming subsequent sequence generation. To address this issue, we propose a novel model regularization method for NMT training, which aims to improve the agreement between translations generated by left-to-right (L2R) and right-to-left (R2L) NMT decoders. This goal is achieved by introducing two Kullback-Leibler divergence regularization terms into the NMT training objective to reduce the mismatch between output probabilities of L2R and R2L models. In addition, we also employ a joint training strategy to allow L2R and R2L models to improve each other in an interactive update process. Experimental results show that our proposed method significantly outperforms state-of-the-art baselines on Chinese-English and English-German translation tasks.

arXiv commentAccepted by AAAI 2019

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Level Training with Recurrent Neural Networks

Cited by in corpus (5)

• Unsupervised Neural Machine Translation with SMT as Posterior Regularization
• Lattice Transformer for Speech Translation
• Synchronous Bidirectional Inference for Neural Sequence Generation
• ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
• Improving Bidirectional Decoding with Dynamic Target Semantics in Neural Machine Translation