Improving LLM-as-a-Judge Inference with the Judgment Distribution

Victor Wang, Michael J. Q. Zhang, Eunsol Choi

Using language models to scalably approximate human preferences on text quality (LLM-as-a-judge) has become a standard practice applicable to many tasks. A judgment is often extrac…