46 citations · 122 across the 12 of their papers we have counts for
14 papers
MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
Daniel Tamayo, Iñaki Lacunza, Paula Rivera-Hidalgo +4
We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model…
XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs
Iñaki Lacunza, José Javier Saiz, Alexander Shvets +2
Current large language models (LLMs) are trained on massive amounts of text data, primarily from a few dominant languages. Studies suggest that this over-reliance on high-resource…
ACADATA: Parallel Dataset of Academic Data for Machine Translation
Iñaki Lacunza, Javier Garcia Gilabert, Francesca De Luca Fornaciari +4
We present ACADATA, a high-quality parallel dataset for academic translation, that consists of two subsets: ACAD-TRAIN, which contains approximately 1.5 million author-generated pa…
EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering
Valle Ruiz-Fernández, Mario Mina, Júlia Falcão +4
Previous literature has largely shown that Large Language Models (LLMs) perpetuate social biases learnt from their pre-training data. Given the notable lack of resources for social…
Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularization
Iñigo Pikabea, Iñaki Lacunza, Oriol Pareras +4
Rapid advancements in Visual Language Models (VLMs) have transformed multimodal understanding but are often constrained by generating English responses regardless of the input lang…
Salamandra Technical Report
Aitor Gonzalez-Agirre, Marc Pàmies, Joan Llop +21
This work introduces Salamandra, a suite of open-source decoder-only large language models available in three different sizes: 2, 7, and 40 billion parameters. The models were trai…
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
Daniel Tamayo, Aitor Gonzalez-Agirre, Javier Hernando +1
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study exp…
Spanish Legalese Language Model and Corpora
Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Aitor Gonzalez-Agirre +1
There are many Language Models for the English language according to its worldwide relevance. However, for the Spanish language, even if it is a widely spoken language, there are v…
Biomedical and Clinical Language Models for Spanish: On the Benefits of Domain-Specific Pretraining in a Mid-Resource Scenario
Casimiro Pio Carrino, Jordi Armengol-Estapé, Asier Gutiérrez-Fandiño +4
This work presents biomedical and clinical language models for Spanish by experimenting with different pretraining choices, such as masking at word and subword level, varying the v…
Spanish Biomedical Crawled Corpus: A Large, Diverse Dataset for Spanish Biomedical Language Models
Casimiro Pio Carrino, Jordi Armengol-Estapé, Ona de Gibert Bonet +4
We introduce CoWeSe (the Corpus Web Salud Español), the largest Spanish biomedical corpus to date, consisting of 4.5GB (about 750M tokens) of clean plain text. CoWeSe is the result…
Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan
Jordi Armengol-Estapé, Casimiro Pio Carrino, Carlos Rodriguez-Penagos +5
Multilingual language models have been a crucial breakthrough as they considerably reduce the need of data for under-resourced languages. Nevertheless, the superiority of language-…
MarIA: Spanish Language Models
Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Marc Pàmies +7
This work presents MarIA, a family of Spanish language models and associated resources made available to the industry and the research community. Currently, MarIA includes RoBERTa-…
Spanish Biomedical and Clinical Language Embeddings
Asier Gutiérrez-Fandiño, Jordi Armengol-Estapé, Casimiro Pio Carrino +3
We computed both Word and Sub-word Embeddings using FastText. For Sub-word embeddings we selected Byte Pair Encoding (BPE) algorithm to represent the sub-words. We evaluated the Bi…
Interpretable Semantic Textual Similarity: Finding and explaining differences between sentences
I. Lopez-Gazpio, M. Maritxalar, A. Gonzalez-Agirre +3
User acceptance of artificial intelligence agents might depend on their ability to explain their reasoning, which requires adding an interpretability layer that fa- cilitates users…