6 papers
Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
Jan-Thorsten Peter, David Vilar, Tobias Domhan +2
Most current large language models (LLMs) support a wide variety of languages in addition to English, including high-resource languages (e.g. German, Chinese, French), as well as l…
You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translation
Gergely Flamich, David Vilar, Jan-Thorsten Peter +1
The goal of translation, be it by human or by machine, is, given some text in a source language, to produce text in a target language that simultaneously 1) preserves the meaning o…
Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translation
Muhammed Yusuf Kocyigit, Eleftheria Briakou, Daniel Deutsch +3
Data contamination -- the accidental consumption of evaluation examples within the pre-training data -- can undermine the validity of evaluation benchmarks. In this paper, we prese…
On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation
Eleftheria Briakou, Zhongtao Liu, Colin Cherry +1
This paper investigates the impact of verbose LLM translations on evaluation. We first demonstrate the prevalence of this behavior across several LLM outputs drawn from the WMT 202…
Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts
Eleftheria Briakou, Jiaming Luo, Colin Cherry +1
In this paper we present a step-by-step approach to long-form text translation, drawing on established processes in translation studies. Instead of viewing machine translation as a…
Assessing biomedical knowledge robustness in large language models by query-efficient sampling attacks
R. Patrick Xian, Alex J. Lee, Satvik Lolla +4
The increasing depth of parametric domain knowledge in large language models (LLMs) is fueling their rapid deployment in real-world applications. Understanding model vulnerabilitie…