Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods
arXiv:2105.03311
Abstract
To facilitate effective translation modeling and translation studies, one of the crucial questions to address is how to assess translation quality. From the perspectives of accuracy, reliability, repeatability and cost, translation quality assessment (TQA) itself is a rich and challenging task. In this work, we present a high-level and concise survey of TQA methods, including both manual judgement criteria and automated evaluation metrics, which we classify into further detailed sub-categories. We hope that this work will be an asset for both translation model researchers and quality assessment researchers. In addition, we hope that it will enable practitioners to quickly develop a better understanding of the conventional TQA field, and to find corresponding closely relevant evaluation solutions for their own needs. This work may also serve inspire further development of quality assessment and evaluation methodologies for other natural language processing (NLP) tasks in addition to machine translation (MT), such as automatic text summarization (ATS), natural language understanding (NLU) and natural language generation (NLG).
arXiv commentAccepted to 23rd Nordic Conference on Computational Linguistics (NoDaLiDa 2021): Workshop on Modelling Translation: Translatology in the Digital Age (MoTra21). arXiv admin note: substantial text overlap with arXiv:1605.04515
References in corpus (6)
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview
- Shellcode_IA32: A Dataset for Automatic Shellcode Generation
- LEPOR: An Augmented Machine Translation Evaluation Metric
- MultiMWE: Building a Multi-lingual Multi-Word Expression (MWE) Parallel Corpora
- Chinese Character Decomposition for Neural MT with Multi-Word Expressions
Cited by in corpus (4)
- EVIL: Exploiting Software via Natural Language
- Can We Generate Shellcodes via Natural Language? An Empirical Study
- Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generation
- cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE