9 papers
Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answers
Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza +2
When large language models (LLMs) are used in high-stakes scenarios, such as legal, medical and financial advice, even a single conversation history is enough to drive differences…
A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents
Raghu Arghal, Fade Chen, Niall Dalton +6
Understanding an agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a fram…
Agents of Chaos
Natalie Shapira, Chris Wendler, Avery Yen +35
We report an exploratory red-teaming study of autonomous language-model-powered agents deployed in a live laboratory environment with persistent memory, email accounts, Discord acc…
Challenging the Abilities of Large Language Models in Italian: a Community Initiative
Malvina Nissim, Danilo Croce, Viviana Patti +78
The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of t…
Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning Evaluation
Khondoker Ittehadul Islam, Gabriele Sarti
Language models have demonstrated remarkable performance on complex multi-step reasoning tasks. However, their evaluation has been predominantly confined to high-resource languages…
Steering Large Language Models for Machine Translation Personalization
Daniel Scalena, Gabriele Sarti, Arianna Bisazza +2
Large language models have simplified the production of personalized translations reflecting predefined stylistic constraints. However, these systems still struggle when stylistic…