MafiaScope: Inspecting Multi-Agent Reasoning in Social Deduction Games

Abstract

An LLM agent's public behaviour reveals little about its social reasoning: an agent that votes correctly may be guessing, and its public claims may differ from privately elicited assessments. We present MafiaScope, an open testbed that turns the social deduction game Mafia into a measurement instrument for multi-agent reasoning. After each public utterance, living agents privately answer configurable structured questions on copies of their available conversations. Probing is non-invasive with respect to the acting context: neither the questions nor the answers enter the histories used to continue the game. Role assessments are scored against roles known to the engine; second-order predictions are compared with other agents' reports. An interactive visualizer renders the assessment trajectories: impersonate mode focuses on one agent's information and reports, panels chart timeline-aligned accuracy and calibration, and counterfactual replay branches from saved states. An offline heuristic links shifts in collective suspicion to episodes for inspection. Experiments span multiple models, languages, and probing configurations; the demonstration presents episode inspection, calibration, probe-repeatability and probe/no-probe checks, and utterance-replacement replay. Code: https://github.com/karpovilia/mafiascope Demo: https://karpovilia.github.io/mafiascope Video: https://vimeo.com/1208920221