19 citations · 19 across the 4 of their papers we have counts for
4 papers · 1 filter
Calibrating LLM Judges for Human and AI Conversations
Maike Züfle, Patrícia Schmidtová, Vilém Zouhar +9
Measuring how successful a conversation is remains difficult, even for humans judging spoken dialogue. We evaluate state-of-the-art LLMs as pointwise and pairwise judges of convers…
Distributional Metrics for Evaluating Spoken Conversational Systems
Shree Harsha Bokkahalli Satish, Erica Cooper, Patrícia Schmidtová +4
Evaluating conversational systems is a difficult and unresolved problem. We introduce the Conversational Distribution Score (CDS), which compares distributions of conversational be…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Pinzhen Chen, Koel Dutta Chowdhury, Xiaoya Xu +20
Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evaluation---a design that is prone…