Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

Paul Resnick, Yuqing Kong, Grant Schoenebeck +1

In many classification tasks, there is no definitive ground truth, only human judgments that may disagree. We address two challenges that arise in such settings: (1) how to use hum…