A Collaborative Mechanism for Crowdsourcing Prediction Problems

Abstract

Machine Learning competitions such as the Netflix Prize have proven reasonably successful as a method of "crowdsourcing" prediction tasks. But these competitions have a number of weaknesses, particularly in the incentive structure they create for the participants. We propose a new approach, called a Crowdsourced Learning Mechanism, in which participants collaboratively "learn" a hypothesis for a given prediction task. The approach draws heavily from the concept of a prediction market, where traders bet on the likelihood of a future event. In our framework, the mechanism continues to publish the current hypothesis, and participants can modify this hypothesis by wagering on an update. The critical incentive property is that a participant will profit an amount that scales according to how much her update improves performance on a released test set.

Full version of the extended abstract which appeared in NIPS 2011

References in corpus (1)

• A Utility Framework for Bounded-Loss Market Makers

Cited by in corpus (10)

• Learning User's confidence for active learning
• Incentives for Federated Learning: a Hypothesis Elicitation Approach
• Collaborative Machine Learning Markets with Data-Replication-Robust Payments
• Machine Learning in/for Blockchain: Future and Challenges
• Designing Informative Securities
• Smooth markets: A basic mechanism for organizing gradient-based learners
• No-Regret and Incentive-Compatible Online Learning
• Aggregating predictions from experts: a scoping review of statistical methods, experiments, and applications
• Semantics, Representations and Grammars for Deep Learning
• Bayesian Regression Markets