Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus

Elias Hossain, Niloofar Yousefi, Ser-Nam Lim

Consensus among LLM judges is often taken as strong evidence that a decision is correct. This assumes that judges make their errors independently. In practice, LLM judges are often…