6 papers
Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation
Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1
Reproducibility is essential for scientific research, yet prior work shows that LLM outputs vary with hardware and batching. We identify an overlooked factor: the hidden injection…
Calibration as a First-Class Criterion in LLM Evaluation
Mario Sanz-Guerrero, Katharina von der Wense
Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it a…
Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions
Minh Duc Bui, Mario Sanz-Guerrero, Abteen Ebrahimi +4
Should multilingual LLMs answer medical questions consistently across input languages, or adapt responses to cultural cues? Existing multilingual medical benchmarks usually assume…
An Investigation of Translationese in the Generations of Multilingual Large Language Models
Maria Valentini, Téa Wright, Julisa Granados +2
Text which has been translated from another language tends to carry with it evidence of translation$\unicode{x2014}$hence, it is often referred to as . Mul…
Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation
Ali Marashian, Alexis Palmer, Katharina von der Wense
The rapid rise in popularity of large language models (LLMs) for translation calls for a thorough study of the reliability of their confidence in their own outputs. Unlike many gen…
It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension
Sagi Shaier, Lawrence E Hunter, Katharina von der Wense
Natural language processing has seen rapid progress over the past decade. Due to the speed of developments, some practices get established without proper evaluation. Considering on…