Tuned Models of Peer Assessment in MOOCs

Abstract

In massive open online courses (MOOCs), peer grading serves as a critical tool for scaling the grading of complex, open-ended assignments to courses with tens or hundreds of thousands of students. But despite promising initial trials, it does not always deliver accurate results compared to human experts. In this paper, we develop algorithms for estimating and correcting for grader biases and reliabilities, showing significant improvement in peer grading accuracy on real data with 63,199 peer grades from Coursera's HCI course offerings --- the largest peer grading networks analysed to date. We relate grader biases and reliabilities to other student factors such as student engagement, performance as well as commenting style. We also show that our model can lead to more intelligent assignment of graders to gradees.

Proceedings of The 6th International Conference on Educational Data Mining (EDM 2013)

References in corpus (1)

• A Framework for Optimizing Paper Matching

Cited by in corpus (26)

• The PeerRank Method for Peer Assessment
• Learning about social learning in MOOCs: From statistical analysis to generative model
• On the Theory and Practice of Privacy-Preserving Bayesian Data Analysis
• MOOCdb: Developing Standards and Systems to Support MOOC Data Science
• Mining MOOC Clickstreams: On the Relationship Between Learner Behavior and Performance
• When is it Better to Compare than to Score?
• Improving essay peer grading accuracy in MOOCs using personalized weights from student's engagement and performance
• Mathematical Language Processing: Automatic Grading and Feedback for Open Response Mathematical Questions
• On Strategyproof Conference Peer Review
• Active Ranking from Pairwise Comparisons and when Parametric Assumptions Don't Help
• Incentive Design in Peer Review: Rating and Repeated Endogenous Matching
• Learning From Graph Neighborhoods Using LSTMs
• TrueReview: A Platform for Post-Publication Peer Review
• Iterative Bayesian Learning for Crowdsourced Regression
• Detecting Problem Statements in Peer Assessments
• Incentives for Truthful Evaluations
• MOOCs Meet Measurement Theory: A Topic-Modelling Approach
• Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms do not Improve over Simple Baselines
• Evaluating Complex Task through Crowdsourcing: Multiple Views Approach
• Robust Mean Estimation on Highly Incomplete Data with Arbitrary Outliers
• Interactive and Iterative Peer Assessment
• Methods for Ordinal Peer Grading
• Online Peer-Assessment Datasets
• Key Phrase Classification in Complex Assignments
• On the Impossibility of Convex Inference in Human Computation
• Educational Content Linking for Enhancing Learning Need Remediation in MOOCs