5 papers
Multilingual GSM-Symbolic: What determines capability transfer across languages?
Kenneth Enevoldsen, Riley Herchert, Sofie Mosegaard +22
We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incomparable, saturation-prone datase…
A multilingual hallucination benchmark: MultiWikiQHalluA
Freja Thoresen, Dan Saattrup Smart
Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness hallucinations, defined as mo…
MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
Sofie Helene Bruun, Dan Saattrup Smart
We create high-quality datasets for LLM evaluation of logical reasoning skills across nine different languages, which have been manually checked by fluent speakers. The datasets co…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages
Dan Saattrup Smart
We introduce a new reading comprehension dataset, dubbed MultiWikiQA, which covers 306 languages and has 1,220,757 samples in total. We start with Wikipedia articles, which also pr…