566 citations · 769 across the 11 of their papers we have counts for
12 papers
Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Hala Almaghout, Christian Federmann, Qin Gao
Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation span…
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data
Haoran Sun, Renren Jin, Shaoyang Xu +10
Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between high- and low-resource lan…
Preliminary WMT24 Ranking of General MT Systems and LLMs
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden +18
This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking…
Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation
Kartik Kartik, Sanjana Soni, Anoop Kunchukuttan +2
The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka code-mixed language) in a single utterance. This…
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Tom Kocmi, Vilém Zouhar, Christian Federmann +1
Ten years ago a single metric, BLEU, governed progress in machine translation research. For better or worse, there is no such consensus today, and consequently it is difficult for…
GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4
Tom Kocmi, Christian Federmann
This paper introduces GEMBA-MQM, a GPT-based evaluation metric designed to detect translation quality errors, specifically for the quality estimation setting without the need for h…
Large Language Models Are State-of-the-Art Evaluators of Translation Quality
Tom Kocmi, Christian Federmann
We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot pro…
Searching for a higher power in the human evaluation of MT
Johnny Tian-Zheng Wei, Tom Kocmi, Christian Federmann
In MT evaluation, pairwise comparisons are conducted to identify the better system. In conducting the comparison, the experimenter must allocate a budget to collect Direct Assessme…
The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task
Shuoyang Ding, Marcin Junczys-Dowmunt, Matt Post +2
This paper presents the JHU-Microsoft joint submission for WMT 2021 quality estimation shared task. We only participate in Task 2 (post-editing effort estimation) of the shared tas…
To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation
Tom Kocmi, Christian Federmann, Roman Grundkiewicz +3
Automatic metrics are commonly used as the exclusive tool for declaring the superiority of one machine translation system's quality over another. The community choice of automatic…
On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs
Roman Grundkiewicz, Marcin Junczys-Dowmunt, Christian Federmann +1
Recent studies emphasize the need of document context in human evaluation of machine translations, but little research has been done on the impact of user interfaces on annotator p…
Achieving Human Parity on Automatic Chinese to English News Translation
Hany Hassan, Anthony Aue, Chang Chen +21
Machine translation has made rapid advances in recent years. Millions of people are using it today in online translation systems and mobile applications in order to communicate acr…