Moral Hazard in Multi-Agent Language Models

Abstract

Cooperation can fail when socially valuable effort is costly, hard to observe, and benefits mainly someone else. Building on Holmström's model of moral hazard in teams, we introduce the Dialogue Moral Hazard Game, a theory-grounded controlled experimental paradigm that instantiates this hidden-action structure as a textual environment for language agents. In each episode, an agent chooses between keeping an immediate local reward and paying a query cost to reveal a hidden safety fact that primarily helps another agent's downstream decision. We evaluate thirteen open-weight and four frontier models with stage-level mechanism metrics. In matched 3,015-decision-per-model experiments, GPT-5.6 Sol and Claude Opus 4.8 track the Holmström-derived private-share boundary across nine query costs (mean absolute errors 0.013 and 0.030); Muse Spark 1.1 responds directionally, whereas Fable 5 remains query-saturated. Diagnostic SFT, RLOO, SFT+RLOO, and GEPA updates are heterogeneous: SmolLM3-3B and OLMo-7B show the clearest weight-level mechanism gains, while GEPA raises Muse team success from $22.2\pm 3.8\%$ to $100.0\pm 0.0\%$ as query use falls from $51.1\pm 5.1\%$ to $0.3\pm 0.5\%$. Freezing the three Muse prompts and intervening on the rank--label mapping changes team success from $100.0\%$ to $12.5\%$ and then $0.0\%$, with validity fixed at $100\%$. Opus supplies a within-model contrast: its query-mediated prompt remains perfect across mappings, while two near-zero-query prompts follow the same trajectory. Optimization can therefore reach the same aggregate outcome through direct revelation or a learned effective information structure, motivating mechanism-level evaluation rather than team success alone.

New frontier baselines, new open weight inference baselines