How To Grade a Test Without Knowing the Answers --- A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing

Abstract

We propose a new probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings. We devise an active learning/adaptive testing scheme based on a greedy minimization of expected model entropy, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on the previous responses. We present experimental results that confirm the ability of our model to infer the required parameters and demonstrate that the adaptive testing scheme requires fewer questions to obtain the same accuracy as a static test scenario.

Appears in Proceedings of the 29th International Conference on Machine Learning (ICML 2012)

References in corpus (1)

• Crowd IQ -- Aggregating Opinions to Boost Performance

Cited by in corpus (8)

• Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing
• Statistical Decision Making for Optimal Budget Allocation in Crowd Labeling
• Graph Mining Meets Crowdsourcing: Extracting Experts for Answer Aggregation
• Detecting Parameter Symmetries in Probabilistic Models
• Soft Label Memorization-Generalization for Natural Language Inference
• Evaluating AI systems under uncertain ground truth: a case study in dermatology
• Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
• TopicResponse: A Marriage of Topic Modelling and Rasch Modelling for Automatic Measurement in MOOCs