7 papers
MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo +4
We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model…
XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs
Iñaki Lacunza, José Javier Saiz, Alexander Shvets +2
Current large language models (LLMs) are trained on massive amounts of text data, primarily from a few dominant languages. Studies suggest that this over-reliance on high-resource…
ACADATA: Parallel Dataset of Academic Data for Machine Translation
Iñaki Lacunza, Javier Garcia Gilabert, Francesca De Luca Fornaciari +4
We present ACADATA, a high-quality parallel dataset for academic translation, that consists of two subsets: ACAD-TRAIN, which contains approximately 1.5 million author-generated pa…
EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering
Valle Ruiz-Fernández, Mario Mina, Júlia Falcão +4
Previous literature has largely shown that Large Language Models (LLMs) perpetuate social biases learnt from their pre-training data. Given the notable lack of resources for social…
Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization
Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras +4
Rapid advancements in Visual Language Models (VLMs) have transformed multimodal understanding but are often constrained by generating English responses regardless of the input lang…
Salamandra Technical Report
Aitor Gonzalez-Agirre, Marc PÃ mies, Joan Llop +21
This work introduces Salamandra, a suite of open-source decoder-only large language models available in three different sizes: 2, 7, and 40 billion parameters. The models were trai…