331 citations · 909 across the 15 of their papers we have counts for
3 papers · 1 filter
Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation
Remi Denton, Mark Díaz, Ian Kivlichan +2
Human annotations play a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into buildin…
Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations
Aida Mostafazadeh Davani, Mark Díaz, Vinodkumar Prabhakaran
Majority voting and averaging are common approaches employed to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators…
On Releasing Annotator-Level Labels and Information in Datasets
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, Mark Díaz
A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then…