How many labelers do you have? A closer look at gold-standard labels

Chen Cheng, Hilal Asi, John Duchi

The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of "gold-standard". W…