Step-Level Preference Learning for Generative Agents in Social Simulations
arXiv:2607.14485
The paper presents an interactive interface to collect step‑level human preference data for generative agents, creates a 57K annotation dataset, and shows that training LLMs with this fine‑grained supervision improves the agents' decision quality and long‑horizon social simulation behavior.
Abstract
Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.
WAICA2026