A Survey on Task Assignment in Crowdsourcing

Abstract

Quality improvement methods are essential to gathering high-quality crowdsourced data, both for research and industry applications. A popular and broadly applicable method is task assignment that dynamically adjusts crowd workflow parameters. In this survey, we review task assignment methods that address: heterogeneous task assignment, question assignment, and plurality problems in crowdsourcing. We discuss and contrast how these methods estimate worker performance, and highlight potential challenges in their implementation. Finally, we discuss future research directions for task assignment methods, and how crowdsourcing platforms and other stakeholders can benefit from them.

arXiv comment36 pages, To appear in ACM Computing Surveys

References in corpus (4)

• Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
• Quizz: Targeted crowdsourcing with a billion (potential) users
• Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms
• Online Assignment of Heterogeneous Tasks in Crowdsourcing Markets

Cited by in corpus (2)

• Design of mechanisms for ensuring the execution of tasks in project planning
• Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent