Human-Paraphrased References Improve Neural Machine Translation

Abstract

Automatic evaluation comparing candidate translations to human-generated paraphrases of reference translations has recently been proposed by Freitag et al. When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment. This effect holds for a variety of different automatic metrics, and tends to favor natural formulations over more literal (translationese) ones. In this paper we compare the results of performing end-to-end system development using standard and paraphrased references. With state-of-the-art English-German NMT components, we show that tuning to paraphrased references produces a system that is significantly better according to human judgment, but 5 BLEU points worse when tested on standard references. Our work confirms the finding that paraphrased references yield metric scores that correlate better with human judgment, and demonstrates for the first time that using these scores for system development can lead to significant improvements.

arXiv commentAccepted at WMT 2020

References in corpus (5)

• BERTScore: Evaluating Text Generation with BERT
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Translationese in Machine Translation Evaluation
• Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation
• Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019

Cited by in corpus (2)

• LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Training
• Language Model Augmented Relevance Score