2 citations · 2 across the 12 of their papers we have counts for
13 papers
MicroVerse: An Instrument for Measuring Self-Authored Identity Drift in Long-Horizon Multi-Agent Language-Model Simulations
Sky Ng, Brihi Joshi, Ishan Gupta +47
Long-horizon, multi-agent language model (LM) simulations are widely proposed for studying social behavior, yet instruments to measure whether persona-conditioned agents maintain i…
PersonaEval: Persona-Based User Simulation for Evaluating Interactive Applications
Yifan Simon Liu, Qianfeng Wen, Yilan Fan +40
Real user studies are important for understanding how people interact with systems under test or already deployed. In practice, however, they are often costly, time-consuming, and…
MatrAIx: Simulating the World with 8.3 Billion Persona Agents
Xiaomin Li, Yuexing Hao, Jianheng Hou +90
Human evaluation of AI systems and digital products is costly, slow, and difficult to scale. Offline evaluations are more scalable but often abstract away human diversity and inter…
PALMs: Using Multi Construct-Grounded Rationales for Modeling Population Preferences in LLMs
Priyanka Dey, Brihi Joshi, Preyashi Poddar +2
Large language models are being extensively used to simulate individual user behavior, yet faithfully representing a population requires capturing the systematic variation in value…
Translating the Untranslatable: An Operationalizable Ontology for Untranslatability
Jacob Bremerman, Brihi Joshi, Hirona Arai +2
Untranslatability, cases where meaning cannot be directly preserved across languages, is well-studied in linguistics but underexplored in NLP. As machine translation (MT) systems i…
Rigorous Interpretation Is a Form of Evaluation
Isabelle Lee, Emmy Liu, Cathy Jiao +4
Current machine learning models are evaluated through behavioral snapshots, with benchmark accuracies, win rates and outcome-based metrics. Model explanations and evaluations, howe…
PrimeX: A Dataset of Worldview, Opinion, and Explanation
Rik Koncel-Kedziorski, Brihi Joshi, Tim Paek
As the adoption of language models advances, so does the need to better represent individual users to the model. Are there aspects of an individual's belief system that a language…
Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations
Keyu He, Tejas Srinivasan, Brihi Joshi +3
When people query Vision-Language Models (VLMs) but cannot see the accompanying visual context (e.g. for blind and low-vision users), augmenting VLM predictions with natural langua…
ELI-Why: Evaluating the Pedagogical Utility of Language Model Explanations
Brihi Joshi, Keyu He, Sahana Ramnath +5
Language models today are widely used in education, yet their ability to tailor responses for learners with varied informational needs and knowledge backgrounds remains under-explo…
Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
Sahana Ramnath, Anurag Mudgil, Brihi Joshi +2
Today, large language models are widely used as judges to evaluate responses from other language models. Hence, it is imperative to benchmark and improve these LLM-judges on real-w…
Improving Language Model Personas via Rationalization with Psychological Scaffolds
Brihi Joshi, Xiang Ren, Swabha Swayamdipta +2
Language models prompted with a user description or persona are being used to predict the user's preferences and opinions. However, existing approaches to building personas mostly…
CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization
Brihi Joshi, Sriram Venkatapathy, Mohit Bansal +2
Evaluating creative text such as human-written stories using language models has always been a challenging task -- owing to the subjectivity of multi-annotator ratings. To mimic th…
OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants
Jaspreet Ranjit, Brihi Joshi, Rebecca Dorn +6
Warning: Contents of this paper may be upsetting. Public attitudes towards key societal issues, expressed on online media, are of immense value in policy and reform efforts, yet ch…