Play to Grade: Testing Coding Games as Classifying Markov Decision Process
arXiv:2110.14615
Abstract
Contemporary coding education often presents students with the task of developing programs that have user interaction and complex dynamic systems, such as mouse based games. While pedagogically compelling, there are no contemporary autonomous methods for providing feedback. Notably, interactive programs are impossible to grade by traditional unit tests. In this paper we formalize the challenge of providing feedback to interactive programs as a task of classifying Markov Decision Processes (MDPs). Each student's program fully specifies an MDP where the agent needs to operate and decide, under reasonable generalization, if the dynamics and reward model of the input MDP should be categorized as correct or broken. We demonstrate that by designing a cooperative objective between an agent and an autoregressive model, we can use the agent to sample differential trajectories from the input MDP that allows a classifier to determine membership: Play to Grade. Our method enables an automatic feedback system for interactive code assignments. We release a dataset of 711,274 anonymized student submissions to a single assignment with hand-coded bug labels to support future research.
NeurIPS 2021, 16 pages, 7 figures
References in corpus (9)
- Deep Knowledge Tracing
- Go-Explore: a New Approach for Hard-Exploration Problems
- Trial without Error: Towards Safe Reinforcement Learning via Human Intervention
- Learning Program Embeddings to Propagate Feedback on Student Code
- Automated Essay Scoring based on Two-Stage Learning
- Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
- The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces
- Generative Grading: Near Human-level Accuracy for Automated Feedback on Richly Structured Problems
- Reliable Deep Grade Prediction with Uncertainty Estimation