Regularized Minimax Conditional Entropy for Crowdsourcing

Abstract

There is a rapidly increasing interest in crowdsourcing for data labeling. By crowdsourcing, a large number of labels can be often quickly gathered at low cost. However, the labels provided by the crowdsourcing workers are usually not of high quality. In this paper, we propose a minimax conditional entropy principle to infer ground truth from noisy crowdsourced labels. Under this principle, we derive a unique probabilistic labeling model jointly parameterized by worker ability and item difficulty. We also propose an objective measurement principle, and show that our method is the only method which satisfies this objective measurement principle. We validate our method through a variety of real crowdsourcing datasets with binary, multiclass or ordinal labels.

31 pages

References in corpus (3)

• Minimax Optimal Convergence Rates for Estimating Ground Truth from Crowdsourced Labels
• Mechanism Design for Crowdsourcing: An Optimal 1-1/e Competitive Budget-Feasible Mechanism for Large Markets
• Adaptive Contract Design for Crowdsourcing Markets: Bandit Algorithms for Repeated Principal-Agent Problems

Cited by in corpus (22)

• Learning From Noisy Singly-labeled Data
• Curriculum Learning for Speech Emotion Recognition from Crowdsourced Labels
• Double or Nothing: Multiplicative Incentive Mechanisms for Crowdsourcing
• A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
• Achieving Budget-optimality with Adaptive Schemes in Crowdsourcing
• Reducing Crowdsourcing to Graphon Estimation, Statistically
• Constrained Labeling for Weakly Supervised Learning
• A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control
• Unlearn What You Have Learned: Adaptive Crowd Teaching with Exponentially Decayed Memory Learners
• Learning from Imperfect Annotations
• Weakly Supervised Label Learning Flows
• Task Recommendation in Crowdsourcing Based on Learning Preferences and Reliabilities
• Practice of Efficient Data Collection via Crowdsourcing at Large-Scale
• Aggregation of pairwise comparisons with reduction of biases
• Iterative Bayesian Learning for Crowdsourced Regression
• Collaborative Data Acquisition
• A Streaming Algorithm for Crowdsourced Data Classification
• Optimal Inference in Crowdsourced Classification via Belief Propagation
• Crowdsourcing with Unsure Option
• Crowdsourcing via Annotator Co-occurrence Imputation and Provable Symmetric Nonnegative Matrix Factorization
• Latent Distribution Assumption for Unbiased and Consistent Consensus Modelling
• Deconvolution with Unknown Error Distribution Interpreted as Blind Isotonic Regression