9 citations · 17 across the 11 of their papers we have counts for
13 papers
Challenging the Abilities of Large Language Models in Italian: a Community Initiative
Malvina Nissim, Danilo Croce, Viviana Patti +78
The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of t…
BERnaT: Basque Encoders for Representing Natural Textual Diversity
Ekhi Azurmendi, Joseba Fernandez de Landa, Jaione Bengoetxea +5
Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robus…
Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque
Lukas Arana, Julen Etxaniz, Ander Salaberria +1
Current Multimodal Large Language Models exhibit very strong performance for several demanding tasks. While commercial MLLMs deliver acceptable performance in low-resource language…
BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data
Jaap Jumelet, Abdellah Fourtassi, Akari Haga +23
We present BabyBabelLM, a multilingual collection of datasets modeling the language a person observes from birth until they acquire a native language. We curate developmentally pla…
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
Oscar Sainz, Naiara Perez, Julen Etxaniz +9
Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages. In this paper, we explore alternatives to…
Truth Knows No Language: Evaluating Truthfulness Beyond English
Blanca Calvo Figueras, Eneko Sagarzazu, Julen Etxaniz +4
We introduce a professionally translated extension of the TruthfulQA benchmark designed to evaluate truthfulness in Basque, Catalan, Galician, and Spanish. Truthfulness evaluations…
HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotation
Jaione Bengoetxea, Mikel Zubillaga, Ekhi Azurmendi +4
In this paper we present our submission for the NorSID Shared Task as part of the 2025 VarDial Workshop (Scherrer et al., 2025), consisting of three tasks: Intent Detection, Slot F…
BertaQA: How Much Do Language Models Know About Local Culture?
Julen Etxaniz, Gorka Azkune, Aitor Soroa +2
Large Language Models (LLMs) exhibit extensive knowledge about the world, but most evaluations have been limited to global or anglocentric subjects. This raises the question of how…
Lessons from the Trenches on Reproducible Evaluation of Language Models
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika +27
Reliable evaluation of language models (LMs) remains an open challenge. Re- searchers and engineers face methodological issues such as the sensitivity of models to evaluation setup…
XNLIeu: a dataset for cross-lingual NLI in Basque
Maite Heredia, Julen Etxaniz, Muitze Zulaika +3
XNLI is a popular Natural Language Inference (NLI) benchmark widely used to evaluate cross-lingual Natural Language Understanding (NLU) capabilities across languages. In this paper…
Latxa: An Open Language Model and Evaluation Suite for Basque
Julen Etxaniz, Oscar Sainz, Naiara Perez +6
We introduce Latxa, a family of large language models for Basque ranging from 7 to 70 billion parameters. Latxa is based on Llama 2, which we continue pretraining on a new Basque c…
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero +3
In this position paper, we argue that the classical evaluation on Natural Language Processing (NLP) tasks using annotated benchmarks is in trouble. The worst kind of data contamina…
Do Multilingual Language Models Think Better in English?
Julen Etxaniz, Gorka Azkune, Aitor Soroa +2
Translate-test is a popular technique to improve the performance of multilingual language models. This approach works by translating the input into English using an external machin…