10 papers
Neyman Orthogonal Activation Steering
David Arbour, Erfan Jahanparast, Serina Chang +1
Activation steering is a popular approach for shifting LLM output toward a desired behavior, and increasingly for generating realistic synthetic data. The standard recipe estimates…
PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards
Yunkai Zhan, Miaosen Chai, Shangshang Wang +5
Reinforcement learning (RL) from unit tests is the common recipe for post-training code models, but a unit test only asks whether the final program passes. We analyze what this uni…
Dataset Signatures in Human-LLM Interactions and User Modeling
Joseph Suh, Serina Chang
Human--LLM interaction datasets shape our understanding of AI use and provide a foundation for downstream research, including training and evaluation of user models. In recent year…
Anamnesis: An Open-Source Platform for Large-Scale Backstory-Conditioned Survey Simulation
Song-Ze Yu, Joseph Suh, Serina Chang +1
We present Anamnesis, an interactive system for demographically controllable survey simulation using large language models. Open-source and designed for non-technical users/researc…
Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses
Jonathan Stray, David Zhai Yang, Steven Luo +2
As AI systems increasingly shape political views, defining and evaluating AI political neutrality is an urgent problem. Here, we propose a new definition of AI political neutrality…
Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
Joseph Suh, Ayush Raj, Minwoo Kang +1
User simulators are increasingly leveraged to build interactive AI assistants, yet how to measure the quality of these simulators remains an open question. In this work, we show ho…
Graph-Based Alternatives to LLMs for Human Simulation
Joseph Suh, Suhong Moon, Serina Chang
Large language models (LLMs) have become a popular approach for simulating human behaviors, yet it remains unclear if LLMs are necessary for all simulation tasks. We study a broad…
Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectification
Stefan Krsteski, Giuseppe Russo, Serina Chang +2
Surveys provide valuable insights into public opinion and behavior, but their execution is costly and slow. Large language models (LLMs) have been proposed as a scalable, low-cost…
ChatBench: From Static Benchmarks to Human-AI Evaluation
Serina Chang, Ashton Anderson, Jake M. Hofman
With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure L…
Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinions
Joseph Suh, Erfan Jahanparast, Suhong Moon +2
Large language models (LLMs) present novel opportunities in public opinion research by predicting survey responses in advance during the early stages of survey design. Prior method…