When Better Codebooks Are Not Enough: Predictive Performance and Behavioral Reliability in LLM Political Event Coding

Abstract

High accuracy does not necessarily make an LLM a faithful coder. This issue matters because many social-science studies rely on expert-written codebooks to turn text into structured data. We study political event coding, where a model must identify the action that one actor directs toward another under detailed coding rules. We compare label names alone with concise definitions and enriched guidance that adds examples, event-mode instructions, and boundary rules. We also evaluate alternative prompting and retrieval methods. We then test behavioral reliability under changes to codebook order, label names, and label-definition mappings. Enriched guidance raises mean root-level macro-F1 from 0.457 to 0.633. Methods with access to definitions remain effective when meaningful label names are removed, but no evaluated method exceeds 0.20 weighted F1 after the label-definition mapping is reassigned. These results motivate separate evaluation of predictive performance and adherence to the supplied coding rules.

13 pages, 3 figures, 13 tables. Revised version with updated experiments, behavioral reliability analyses, and additional API-model results