Rho-Perfect: Correlation Ceiling For Subjective Evaluation Datasets

Fredrik Cumlin

Subjective ratings contain inherent noise that limits the model-human correlation, but this reliability issue is rarely quantified. In this paper, we present $\rho$-Perfect, a practic…