Greedy Search with Probabilistic N-gram Matching for Neural Machine Translation
arXiv:1809.03132
Abstract
Neural machine translation (NMT) models are usually trained with the word-level loss using the teacher forcing algorithm, which not only evaluates the translation improperly but also suffers from exposure bias. Sequence-level training under the reinforcement framework can mitigate the problems of the word-level loss, but its performance is unstable due to the high variance of the gradient estimation. On these grounds, we present a method with a differentiable sequence-level training objective based on probabilistic n-gram matching which can avoid the reinforcement framework. In addition, this method performs greedy search in the training which uses the predicted words as context just as at inference to alleviate the problem of exposure bias. Experiment results on the NIST Chinese-to-English translation tasks show that our method significantly outperforms the reinforcement-based algorithms and achieves an improvement of 1.5 BLEU points on average over a strong baseline system.
arXiv comment7 pages, accepted by emnlp 2018
References in corpus (14)
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- ADADELTA: An Adaptive Learning Rate Method
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Convolutional Sequence to Sequence Learning
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
- Sequence Level Training with Recurrent Neural Networks
- Dual Learning for Machine Translation
- Achieving Human Parity on Automatic Chinese to English News Translation
- Professor Forcing: A New Algorithm for Training Recurrent Networks
- Phrase-Based & Neural Unsupervised Machine Translation
- An Actor-Critic Algorithm for Sequence Prediction
- Adversarial Neural Machine Translation
- Bag-of-Words as Target for Neural Machine Translation
Cited by in corpus (5)
- Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
- Modeling Fluency and Faithfulness for Diverse Neural Machine Translation
- Sequence-Level Training for Non-Autoregressive Neural Machine Translation
- Improving Text Generation with Student-Forcing Optimal Transport
- Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation