Quantifying and Correcting Measurement Error in LLM-Generated Classifications: A Beta-Binomial Latent Class Model for Correlated Repeated Judgments

Yichi Zhang, Ignacio Martinez

Large language models are increasingly used to classify text at scale. Because their outputs are stochastic, it is common to query a model several times per item and aggregate the…