Bias and Uncertainty in LLM-as-a-Judge Estimation

James Fiedler

LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator, i.e., raw judge outputs, is…