Large Language Models Are State-of-the-Art Evaluators of Translation Quality

Abstract

We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.

Accepted in EAMT, 10 pages, 8 tables, one figure

Cited by in corpus (8)

• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
• WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia
• From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation
• Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language models
• Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
• Generating Faithful Text From a Knowledge Graph with Noisy Reference Text