Translationese in Machine Translation Evaluation

Abstract

The term translationese has been used to describe the presence of unusual features of translated text. In this paper, we provide a detailed analysis of the adverse effects of translationese on machine translation evaluation results. Our analysis shows evidence to support differences in text originally written in a given language relative to translated text and this can potentially negatively impact the accuracy of machine translation evaluations. For this reason we recommend that reverse-created test data be omitted from future machine translation test sets. In addition, we provide a re-evaluation of a past high-profile machine translation evaluation claiming human-parity of MT, as well as analysis of the since re-evaluations of it. We find potential ways of improving the reliability of all three past evaluations. One important issue not previously considered is the statistical power of significance tests applied in past evaluations that aim to investigate human-parity of MT. Since the very aim of such evaluations is to reveal legitimate ties between human and MT systems, power analysis is of particular importance, where low power could result in claims of human parity that in fact simply correspond to Type II error. We therefore provide a detailed power analysis of tests used in such evaluations to provide an indication of a suitable minimum sample size of translations for such studies. Subsequently, since no past evaluation that aimed to investigate claims of human parity ticks all boxes in terms of accuracy and reliability, we rerun the evaluation of the systems claiming human parity. Finally, we provide a comprehensive check-list for future machine translation evaluation.

17 pages, 8 figures, 9 tables

Cited by in corpus (15)

• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
• Translation Artifacts in Cross-lingual Transfer Learning
• Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation
• Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019
• A Survey on Document-level Neural Machine Translation: Methods and Evaluation
• Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation
• Human-Paraphrased References Improve Neural Machine Translation
• Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns
• Language Models are Good Translators
• Self-Training Sampling with Monolingual Data Uncertainty for Neural Machine Translation
• NEJM-enzh: A Parallel Corpus for English-Chinese Translation in the Biomedical Domain
• Pronoun-Targeted Fine-tuning for NMT with Hybrid Losses
• Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation
• Document Sub-structure in Neural Machine Translation
• How Furiously Can Colourless Green Ideas Sleep? Sentence Acceptability in Context