When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Zongyou Yang, Yinghan Hou, Xiaokun Yang

An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measuremen…