8 papers
Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions
Dennis Fucci, Andrea Bacciu, Dong Liu +2
LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is…
Data-Driven Personas for Survey Simulation: Insights into Simulation Alignment Across Data-Access Regimes
Dongryeol Lee, Weronika Łajewska, Leonardo Perelli +1
Large language models (LLMs) offer new opportunities for public opinion research by enabling early prediction of survey responses, potentially reducing the cost and time of traditi…
Augmented Hypothesis Testing with Persona-Based LLM Simulations
Ziyad Benomar, Aymen Al Marjani, Paul Missault +1
A/B testing requires large sample sizes, long timelines, and significant costs. When auxiliary predictions of experimental outcomes are available from machine learning models, unce…
Data-Driven Persona-Conditioned Agents for A/B Test Simulation
Ziyad Benomar, Weronika Łajewska, Leonardo Perelli +1
A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulati…
Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Andrea Alfarano, Andrea Bacciu, Saab Mansour +2
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale e…
RADIUS: Ranking, Distribution, and Significance - A Comprehensive Alignment Suite for Survey Simulation
Weronika Łajewska, Paul Missault, George Davidson +1
Simulation of surveys using LLMs is emerging as a powerful application for generating human-like responses at scale. Prior work evaluates survey simulation using metrics borrowed f…
Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition
Ivaxi Sheth, Zeno Jonke, Amin Mantrach +1
As large language models are increasingly deployed across diverse real-world applications, extending automated evaluation beyond English has become a critical challenge. Existing e…
The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation
Aparna Elangovan, Lei Xu, Mahsa Elyasi +8
Benchmarking the capabilities of AI systems, including Large Language Models (LLMs) and Vision Models, typically ignores the impact of uncertainty in the underlying ground truth an…