Crowd Guilds: Worker-led Reputation and Feedback on Crowdsourcing Platforms

Abstract

Crowd workers are distributed and decentralized. While decentralization is designed to utilize independent judgment to promote high-quality results, it paradoxically undercuts behaviors and institutions that are critical to high-quality work. Reputation is one central example: crowdsourcing systems depend on reputation scores from decentralized workers and requesters, but these scores are notoriously inflated and uninformative. In this paper, we draw inspiration from historical worker guilds (e.g., in the silk trade) to design and implement crowd guilds: centralized groups of crowd workers who collectively certify each other's quality through double-blind peer assessment. A two-week field experiment compared crowd guilds to a traditional decentralized crowd work model. Crowd guilds produced reputation signals more strongly correlated with ground-truth worker quality than signals available on current crowd working platforms, and more accurate than in the traditional model.

12 pages, 6 figures, 1 table. To be presented at CSCW2017

References in corpus (4)

• Thousands of Positive Reviews: Distributed Mentoring in Online Fan Communities
• Boomerang: Rebounding the Consequences of Reputation Feedback on Crowdsourcing Platforms
• A Glimpse Far into the Future: Understanding Long-term Crowd Worker Quality
• Atelier: Repurposing Expert Crowdsourcing Tasks as Micro-internships

Cited by in corpus (5)

• Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
• A Survey on Task Assignment in Crowdsourcing
• Crowd Coach: Peer Coaching for Crowd Workers' Skill Growth
• Belief Miner: A Methodology for Discovering Causal Beliefs and Causal Illusions from General Populations
• The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help