30 citations · 94 across the 30 of their papers we have counts for
22 papers · 1 filter
Decoding-Level Taboo: A Diagnostic Stress Test for LLM Robustness
Tadanobu Chuyo Kamijo, Ori Rottenstreich, Javier Conde +2
Large language model evaluations typically focus on performance under nominal conditions, creating an illusion of capability where models comfortably walk a narrow, highly optimize…
Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
Samer Awad, Javier Conde, Carlos Arriaga +3
Modern Large Language Models (LLMs) are often criticized for producing repetitive and homogeneous text, despite possessing vast latent vocabularies. While previous research has foc…
To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times
Thomas Hikaru Clark, Carlos Arriaga, Javier Conde +2
Large Language Models (LLMs) have recently been shown to produce estimates of psycholinguistic norms, such as valence, arousal, or concreteness, for words and multiword expressions…
Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?
Tairan Fu, Javier Conde, Pedro Reviriego +3
Summarization is a core task in Natural Language Processing (NLP). Recent advances in Large Language Models (LLMs) and the introduction of large context windows reaching millions o…
How does fine-tuning improve sensorimotor representations in large language models?
Minghua Wu, Javier Conde, Pedro Reviriego +1
Large Language Models (LLMs) exhibit a significant "embodiment gap", where their text-based representations fail to align with human sensorimotor experiences. This study systematic…
Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings
Javier Conde, María Grandury, Tairan Fu +7
Word-level psycholinguistic norms lend empirical support to theories of language processing. However, obtaining such human-based measures is not always feasible or straightforward.…
La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America
María Grandury, Javier Aula-Blasco, Júlia Falcão +22
Leaderboards showcase the current capabilities and limitations of Large Language Models (LLMs). To motivate the development of LLMs that represent the linguistic and cultural diver…
Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
Raquel Ferrando, Javier Conde, Gonzalo Martínez +1
The computational and energy costs of Large Language Models (LLMs) have increased exponentially driven by the growing model sizes and the massive adoption of LLMs by hundreds of mi…
Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans
Javier Conde, Miguel González, María Grandury +3
The evaluation of LLMs has so far focused primarily on how well they can perform different tasks such as reasoning, question-answering, paraphrasing, or translating. For most of th…
Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?
Tairan Fu, Miguel González, Javier Conde +2
Multimodal Large Language Models which can answer complex questions on an image struggle to tell the time on analog clocks. This is probably due to the lack of images with clocks a…
Speed and Conversational Large Language Models: Not All Is About Tokens per Second
Javier Conde, Miguel González, Pedro Reviriego +3
The speed of open-weights large language models (LLMs) and its dependency on the task at hand, when run on GPUs, is studied to present a comparative analysis of the speed of the mo…
Can ChatGPT Learn to Count Letters?
Javier Conde, Gonzalo Martínez, Pedro Reviriego +3
Large language models (LLMs) struggle on simple tasks such as counting the number of occurrences of a letter in a word. In this paper, we investigate if ChatGPT can learn to count…
Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
Tairan Fu, Javier Conde, Gonzalo Martínez +2
Multiple Choice Question (MCQ) tests are among the most used methods for evaluating large language models (LLMs). Besides checking the correctness of the selected answer, evaluatio…
Why Do Large Language Models (LLMs) Struggle to Count Letters?
Tairan Fu, Raquel Ferrando, Javier Conde +2
Large Language Models (LLMs) have achieved unprecedented performance on many complex tasks, being able, for example, to answer questions on almost any topic. However, they struggle…
Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?
Marina Mayor-Rocher, Nina Melero, Elena Merino-Gómez +3
Large Language Models (LLMs) have been profusely evaluated on their ability to answer questions on many topics and their performance on different natural language understanding tas…
Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal
Gonzalo Martínez, Juan Diego Molero, Sandra González +3
This study investigates the potential of large language models (LLMs) to provide accurate estimates of concreteness, valence and arousal for multi-word expressions. Unlike previous…
Spanish and LLM Benchmarks: is MMLU Lost in Translation?
Irene Plaza, Nina Melero, Cristina del Pozo +4
The evaluation of Large Language Models (LLMs) is a key element in their continuous improvement process and many benchmarks have been developed to assess the performance of LLMs in…
Open Conversational LLMs do not know most Spanish words
Javier Conde, Miguel González, Nina Melero +5
The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-s…
Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study
Gonzalo Martínez, José Alberto Hernández, Javier Conde +2
The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasonin…
Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models
Gonzalo Martínez, Javier Conde, Elena Merino-Gómez +4
Vocabulary tests, once a cornerstone of language modeling evaluation, have been largely overlooked in the current landscape of Large Language Models (LLMs) like Llama, Mistral, and…
How many words does ChatGPT know? The answer is ChatWords
Gonzalo Martínez, Javier Conde, Pedro Reviriego +3
The introduction of ChatGPT has put Artificial Intelligence (AI) Natural Language Processing (NLP) in the spotlight. ChatGPT adoption has been exponential with millions of users ex…
Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
Pedro Reviriego, Javier Conde, Elena Merino-Gómez +2
The introduction of Artificial Intelligence (AI) generative language models such as GPT (Generative Pre-trained Transformer) and tools such as ChatGPT has triggered a revolution th…