6 papers
Does Every Instance Count Equally? Order Effects in LLM Multi-Instance Processing
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
Users increasingly rely on large language models (LLMs) to analyse collections of independent inputs, such as product reviews or social media posts, and return a single aggregated…
Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media
Yuefeng Shi, Nedjma Ousidhoum, Jose Camacho-Collados
LLMs have demonstrated exceptional proficiency in a wide range of NLP tasks. However, a notable gap remains in practical data analysis scenarios, particularly when LLMs are require…
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros +27
We present our shared task on evaluating the adaptability of LLMs and NLP systems across multiple languages and cultures. The task data consist of an extended version of our manual…
Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length
Jingxuan Chen, Mohammad Taher Pilehvar, Jose Camacho-Collados
Users often rely on Large Language Models (LLMs) for processing multiple documents or performing analysis over a number of instances. For example, analysing the overall sentiment o…
Why are all LLMs Obsessed with Japanese Culture? On the Hidden Cultural and Regional Biases of LLMs
Joseba Fernandez de Landa, Carla Perez-Almendros, Jose Camacho-Collados
LLMs have limitations when it comes to cultural coverage and competence, and in some cases, show specific cultural biases. Although prior studies have examined the cultural capabil…
Report-based Recommendations for Policy Making and Agency Operations: Dataset and LLM Evaluation
Aleksandra Edwards, Thomas Edwards, Jose Camacho-Collados +1
Large Language Models (LLMs) are extensively used in text generation tasks. These generative capabilities bring us to a point where LLMs could potentially provide useful insights i…