JudgeProfile: Understanding and Steering Subjectivity in LLM Judges

Qi Cao, Kangning Liu, Xuan Kan +10

LLM judges are inherently subjective, often favoring different responses in pairwise comparison when neither option is objectively wrong. To study this subjectivity, we introduce J…