Reliable Crowdsourcing for Multi-Class Labeling using Coding Theory

Abstract

Crowdsourcing systems often have crowd workers that perform unreliable work on the task they are assigned. In this paper, we propose the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification despite unreliable crowd workers. Coding-theory based techniques also allow us to pose easy-to-answer binary questions to the crowd workers. We consider three different crowdsourcing models: systems with independent crowd workers, systems with peer-dependent reward schemes, and systems where workers have common sources of information. For each of these models, we analyze classification performance with the proposed coding-based scheme. We develop an ordering principle for the quality of crowds and describe how system performance changes with the quality of the crowd. We also show that pairing among workers and diversification of the questions help in improving system performance. We demonstrate the effectiveness of the proposed coding-based scheme using both simulated data and real datasets from Amazon Mechanical Turk, a crowdsourcing microtask platform. Results suggest that use of good codes may improve the performance of the crowdsourcing task over typical majority-voting approaches.

20 pages, 11 figures, under revision, IEEE Journal of Selected Topics in Signal Processing

References in corpus (1)

• Sailing the Information Ocean with Awareness of Currents: Discovery and Application of Source Dependence

Cited by in corpus (14)

• Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
• Emerging Privacy Issues and Solutions in Cyber-Enabled Sharing Services: From Multiple Perspectives
• Prospect Theory Based Crowdsourcing for Classification in the Presence of Spammers
• Multi-object Classification via Crowdsourcing with a Reject Option
• Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?
• A Near-Optimal Change-Detection Based Algorithm for Piecewise-Stationary Combinatorial Semi-Bandits
• Global Multiclass Classification and Dataset Construction via Heterogeneous Local Experts
• Universal Clustering via Crowdsourcing
• Distributed Inference in Tree Networks using Coding Theory
• Don't Fear the Bit Flips: Optimized Coding Strategies for Binary Classification
• Decision Tree Design for Classification in Crowdsourcing Systems
• On Multiple-Access in Queue-Length Sensitive Systems
• Decision Making in Star Networks with Incorrect Beliefs
• Capacity-achieving Polar-based LDGM Codes