Balanced Accuracy: The Right Metric for Evaluating LLM Judges -- Explained through Youden's J statistic

Stephane Collot, Colin Fraser, Justin Zhao +3

Rigorous evaluation of large language models (LLMs) relies on comparing models by the prevalence of desirable or undesirable behaviors, such as task pass rates or policy violations…