Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

Anissa Alloula, Federico Licini, Ava Batchkala +1

LLMs-as-judges are the primary way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchm…