Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory

Fabian David Schmidt, Donato Crisostomi, Carlos Lassance +1

Rerankers decide which documents users and LLMs see, yet their standard metric, nDCG, relies on human relevance labels that are costly, sparse, noisy, and discretely graded. As rer…