Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Abstract

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.

40 pages main paper, 5 pages appendix

References in corpus (8)

• Incentivizing High Quality Crowdwork
• Information filtering via Iterative Refinement
• Near-Optimally Teaching the Crowd to Classify
• Analytic Methods for Optimizing Realtime Crowdsourcing
• Decoding Information from noisy, redundant, and intentionally-distorted sources
• Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
• TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings
• When majority voting fails: Comparing quality assurance methods for noisy human computation environment

Cited by in corpus (18)

• Showing Academic Performance Predictions during Term Planning: Effects on Students' Decisions, Behaviors, and Preferences
• Perspectives on Large Language Models for Relevance Judgment
• Decentralized & Collaborative AI on Blockchain
• A Survey on Task Assignment in Crowdsourcing
• Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspective
• Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing
• Detecting The Corruption Of Online Questionnaires By Artificial Intelligence
• If in a Crowdsourced Data Annotation Pipeline, a GPT-4
• A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification
• In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review
• Synthesizing Event-centric Knowledge Graphs of Daily Activities Using Virtual Space
• Towards Filling the Gap in Conversational Search: From Passage Retrieval to Conversational Response Generation
• Automatically Labeling Low Quality Content on Wikipedia by Leveraging Patterns in Editing Behaviors
• Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data
• The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help
• Beyond Community Notes: A Framework for Understanding and Building Crowdsourced Context Systems for Social Media
• A Labeling Task Design for Supporting Algorithmic Needs: Facilitating Worker Diversity and Reducing AI Bias
• CoSight: Exploring Viewer Contributions to Online Video Accessibility Through Descriptive Commenting