CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

Abstract

Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.

11 pages, Accepted at 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT). arXiv admin note: text overlap with arXiv:2112.04554

References in corpus (3)

• Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
• Quantifying the Invisible Labor in Crowd Work
• Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?

Cited by in corpus (14)

• "HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media
• AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
• Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
• Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention
• A Scoping Study of Evaluation Practices for Responsible AI Tools: Steps Towards Effectiveness Evaluations
• If in a Crowdsourced Data Annotation Pipeline, a GPT-4
• A domain-specific language for describing machine learning datasets
• Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways
• Saliency Cards: A Framework to Characterize and Compare Saliency Methods
• Making Intelligence: Ethical Values in IQ and ML Benchmarks
• Auditing Cross-Cultural Consistency of Human-Annotated Labels for Recommendation Systems
• Evaluating the Social Impact of Generative AI Systems in Systems and Society
• From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gap
• Talking About the Assumption in the Room