57 citations · 63 across the 5 of their papers we have counts for
5 papers
D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation
Aida Mostafazadeh Davani, Mark Díaz, Dylan Baker +1
While human annotations play a crucial role in language technologies, annotator subjectivity has long been overlooked in data collection. Recent studies that have critically examin…
Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation
Andrew Smart, Ding Wang, Ellis Monk +4
Data annotation remains the sine qua non of machine learning and AI. Recent empirical work on data annotation has begun to highlight the importance of rater diversity for fairness,…
The illusion of artificial inclusion
William Agnew, A. Stevie Bergman, Jennifer Chien +5
Human participants play a central role in the development of modern artificial intelligence (AI) technology, in psychological science, and in user research. Recent advances in gene…
(Re)Defining Expertise in Machine Learning Development
Mark Díaz, Angela D. R. Smith
Domain experts are often engaged in the development of machine learning systems in a variety of ways, such as in data collection and evaluation of system performance. At the same t…
The Reasonable Effectiveness of Diverse Evaluation Data
Lora Aroyo, Mark Diaz, Christopher Homan +3
In this paper, we present findings from an semi-experimental exploration of rater diversity and its influence on safety annotations of conversations generated by humans talking to…