5 papers
TatBLiMP: A Benchmark of Linguistic Minimal Pairs for Tatar
Ilshat Saetov, Dmitry Gaynullin
We introduce TatBLiMP, the first benchmark of linguistic minimal pairs for Tatar (tt, ISO 639-3 tat), a Qypchaq Turkic language written in Cyrillic. To our knowledge it is the firs…
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Sherzod Hakimov, Karl Osswald, Jelle Psurek +3
We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unl…
CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data
Pedro Ortiz Suarez, Laurie Burchell, Catherine Arnett +94
Language identification (LID) is a fundamental step in curating multilingual corpora. However, LID models still perform poorly for many languages, especially on the noisy and heter…
Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
Michael Hoffmann, Jophin John, Stefan Schweter +5
We present Llama-GENBA-10B, a trilingual foundation model addressing English-centric bias in large language models. Built on Llama 3.1-8B and scaled to 10B parameters, Llama-GENBA-…
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili +13
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However,…