5 papers
Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+
Sherzod Hakimov, Karl Osswald, Jelle Psurek +3
We evaluate large language models (LLMs) as language agents playing goal-directed dialogue games in self-play across 30 languages: the 24 official EU languages plus six others. Unl…
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler A. Chang, Catherine Arnett, Abdelrahman Sadallah +377
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we pre…
TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages
Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili +13
Being able to thoroughly assess massive multi-task language understanding (MMLU) capabilities is essential for advancing the applicability of multilingual language models. However,…
Enhanced document retrieval with topic embeddings
Kavsar Huseynova, Jafar Isbarov
Document retrieval systems have experienced a revitalized interest with the advent of retrieval-augmented generation (RAG). RAG architecture offers a lower hallucination rate than…
Open foundation models for Azerbaijani language
Jafar Isbarov, Kavsar Huseynova, Elvin Mammadov +2
The emergence of multilingual large language models has enabled the development of language understanding and generation systems in Azerbaijani. However, most of the production-gra…