15 citations · 48 across the 12 of their papers we have counts for
12 papers · 1 filter
A Critical Study of Automatic Evaluation in Sign Language Translation
Shakib Yazdani, Yasser Hamidullah, Cristina España-Bonet +2
Automatic evaluation metrics are crucial for advancing sign language translation (SLT). Current SLT evaluation metrics, such as BLEU and ROUGE, are only text-based, and it remains…
Preliminary Ranking of WMT25 General Machine Translation Systems
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden +25
We present the preliminary rankings of machine translation (MT) systems submitted to the WMT25 General Machine Translation Shared Task, as determined by automatic evaluation metric…
The TUB Sign Language Corpus Collection
Eleftherios Avramidis, Vera Czehmann, Fabian Deckert +8
We present a collection of parallel corpora of 12 sign languages in video format, together with subtitles in the dominant spoken languages of the corresponding countries. The entir…
Evaluation of a Sign Language Avatar on Comprehensibility, User Experience \& Acceptability
Fenya Wasserroth, Eleftherios Avramidis, Vera Czehmann +3
This paper presents an investigation into the impact of adding adjustment features to an existing sign language (SL) avatar on a Microsoft Hololens 2 device. Through a detailed ana…
Preliminary WMT24 Ranking of General MT Systems and LLMs
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden +18
This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking…
Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis +5
High-quality Machine Translation (MT) evaluation relies heavily on human judgments. Comprehensive error classification methods, such as Multidimensional Quality Metrics (MQM), are…
Fine-grained linguistic evaluation for state-of-the-art Machine Translation
Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel +2
This paper describes a test suite submission providing detailed statistics of linguistic performance for the state-of-the-art German-English systems of the Fifth Conference of Mach…
Fine-grained evaluation of Quality Estimation for Machine translation based on a linguistically-motivated Test Suite
Avramidis Eleftherios, Vivien Macketanz, Arle Lommel +1
We present an alternative method of evaluating Quality Estimation systems, which is based on a linguistically-motivated Test Suite. We create a test-set consisting of 14 linguistic…
Fine-grained evaluation of German-English Machine Translation based on a Test Suite
Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt +1
We present an analysis of 16 state-of-the-art MT systems on German-English based on a linguistically-motivated test suite. The test suite has been devised manually by a team of lan…
Linguistic evaluation of German-English Machine Translation using a Test Suite
Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel +1
We present the results of the application of a grammatical test suite for GermanEnglish MT on the systems submitted at WMT19, with a detailed analysis for 107 phenomen…
Train, Sort, Explain: Learning to Diagnose Translation Models
Robert Schwarzenberg, David Harbecke, Vivien Macketanz +2
Evaluating translation models is a trade-off between effort and detail. On the one end of the spectrum there are automatic count-based methods such as BLEU, on the other end lingui…
Sentence-level quality estimation by predicting HTER as a multi-component metric
Eleftherios Avramidis
This submission investigates alternative machine learning models for predicting the HTER score on the sentence level. Instead of directly predicting the HTER score, we suggest a mo…