TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings
arXiv:1206.4606
Abstract
This paper revisits the problem of analyzing multiple ratings given by different judges. Different from previous work that focuses on distilling the true labels from noisy crowdsourcing ratings, we emphasize gaining diagnostic insights into our in-house well-trained judges. We generalize the well-known DawidSkene model (Dawid & Skene, 1979) to a spectrum of probabilistic models under the same "TrueLabel + Confusion" paradigm, and show that our proposed hierarchical Bayesian model, called HybridConfusion, consistently outperforms DawidSkene on both synthetic and real-world data sets.
ICML2012
Cited by in corpus (8)
- Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
- Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
- Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model
- CrowdRev: A platform for Crowd-based Screening of Literature Reviews
- Context-aware learning for generative models
- A Technical Survey on Statistical Modelling and Design Methods for Crowdsourcing Quality Control
- A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
- Active Learning from Crowd in Document Screening