Robust AI Evaluation through Maximal Lotteries

Hadi Khalaf, Serena L. Wang, Daniel Halpern +3

The standard way to evaluate language models on subjective tasks is through pairwise comparisons: an annotator chooses the "better" of two responses to a prompt. Leaderboards aggre…