Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing

Abstract

Crowdsourcing has gained immense popularity in machine learning applications for obtaining large amounts of labeled data. Crowdsourcing is cheap and fast, but suffers from the problem of low-quality data. To address this fundamental challenge in crowdsourcing, we propose a simple payment mechanism to incentivize workers to answer only the questions that they are sure of and skip the rest. We show that surprisingly, under a mild and natural "no-free-lunch" requirement, this mechanism is the one and only incentive-compatible payment mechanism possible. We also show that among all possible incentive-compatible mechanisms (that may or may not satisfy no-free-lunch), our mechanism makes the smallest possible payment to spammers. We further extend our results to a more general setting in which workers are required to provide a quantized confidence for each question. Interestingly, this unique mechanism takes a "multiplicative" form. The simplicity of the mechanism is an added benefit. In preliminary experiments involving over 900 worker-task pairs, we observe a significant drop in the error rates under this unique mechanism for the same or lower monetary expenditure.

References in corpus (6)

• Simple, Robust and Optimal Ranking from Pairwise Comparisons
• Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence
• Regularized Minimax Conditional Entropy for Crowdsourcing
• A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
• Prediction Markets, Mechanism Design, and Cooperative Game Theory
• Approval Voting and Incentives in Crowdsourcing

Cited by in corpus (13)

• A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
• Approval Voting and Incentives in Crowdsourcing
• Prospect Theory Based Crowdsourcing for Classification in the Presence of Spammers
• Does Confidence Reporting from the Crowd Benefit Crowdsourcing Performance?
• A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control
• ZebraLancer: Decentralized Crowdsourcing of Human Knowledge atop Open Blockchain
• An Incentive Mechanism for Crowd Sensing with Colluding Agents
• Avoiding Imposters and Delinquents: Adversarial Crowdsourcing and Peer Prediction
• Efficient Crowdsourcing via Proxy Voting
• Crowdsourcing with Unsure Option
• Recommendation Systems and Self Motivated Users
• Equilibrium Selection in Data Markets: Multiple-Principal, Multiple-Agent Problems with Non-Rivalrous Goods
• Working in Pairs: Understanding the Effects of Worker Interactions in Crowdwork