"All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations

Michael Hardy

"Gold" and "ground truth" human-mediated labels have error. The effects of this error can escape commonly reported metrics of label quality or obscure questions of accuracy, bias,…