Reliable but Design-Sensitive: Instrument Uncertainty in LLM Annotation

Thomas Reiter, Christoph Kern, Fedor Miasnikov +4

Large language models (LLMs) can give reliable labels under one setup yet change those labels when researchers make other reasonable design choices. We tested seven LLMs, 12 task d…