Beyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter

Abstract

Neural machine translation usually adopts autoregressive models and suffers from exposure bias as well as the consequent error propagation problem. Many previous works have discussed the relationship between error propagation and the \emph{accuracy drop} (i.e., the left part of the translated sentence is often better than its right part in left-to-right decoding models) problem. In this paper, we conduct a series of analyses to deeply understand this problem and get several interesting findings. (1) The role of error propagation on accuracy drop is overstated in the literature, although it indeed contributes to the accuracy drop problem. (2) Characteristics of a language play a more important role in causing the accuracy drop: the left part of the translation result in a right-branching language (e.g., English) is more likely to be more accurate than its right part, while the right part is more accurate for a left-branching language (e.g., Japanese). Our discoveries are confirmed on different model structures including Transformer and RNN, and in other sequence generation tasks such as text summarization.

arXiv commentEMNLP 2018

References in corpus (9)

• Sequence to Sequence Learning with Neural Networks
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Sequence Level Training with Recurrent Neural Networks
• Dual Learning for Machine Translation
• Achieving Human Parity on Automatic Chinese to English News Translation
• A Closer Look at Memorization in Deep Networks
• Adversarial Neural Machine Translation
• A Study of Reinforcement Learning for Neural Machine Translation
• Double Path Networks for Sequence to Sequence Learning

Cited by in corpus (6)

• A Study of Reinforcement Learning for Neural Machine Translation
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
• Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
• Improving Neural Machine Translation with Pre-trained Representation
• Neural Machine Translation with Error Correction