Generative Grading: Near Human-level Accuracy for Automated Feedback on Richly Structured Problems

Abstract

Access to high-quality education at scale is limited by the difficulty of providing student feedback on open-ended assignments in structured domains like computer programming, graphics, and short response questions. This problem has proven to be exceptionally difficult: for humans, it requires large amounts of manual work, and for computers, until recently, achieving anything near human-level accuracy has been unattainable. In this paper, we present generative grading: a novel computational approach for providing feedback at scale that is capable of accurately grading student work and providing nuanced, interpretable feedback. Our approach uses generative descriptions of student cognition, written as probabilistic programs, to synthesise millions of labelled example solutions to a problem; we then learn to infer feedback for real student solutions based on this cognitive model. We apply our methods to three settings. In block-based coding, we achieve a 50% improvement upon the previous best results for feedback, achieving super-human accuracy. In two other widely different domains -- graphical tasks and short text answers -- we achieve major improvement over the previous state of the art by about 4x and 1.5x respectively, approaching human accuracy. In a real classroom, we ran an experiment where we used our system to augment human graders, yielding doubled grading accuracy while halving grading time.

arXiv comment10 pages of content

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An efficient, multiple range random walk algorithm to calculate the density of states
• Deep Knowledge Tracing
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Automated Essay Scoring based on Two-Stage Learning
• Deep Amortized Inference for Probabilistic Programs
• Zero Shot Learning for Code Education: Rubric Sampling with Deep Learning Inference
• Reliable Deep Grade Prediction with Uncertainty Estimation
• Leveraging Quality Prediction Models for Automatic Writing Feedback

Cited by in corpus (4)

• The Death of the Short-Form Physics Essay in the Coming AI Revolution
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
• Play to Grade: Testing Coding Games as Classifying Markov Decision Process
• Automating Program Structure Classification