Assessing Confidence in AI-Assisted Grading of Physics Exams through Psychometrics: An Exploratory Study

Abstract

This study explores the use of artificial intelligence in grading high-stakes physics exams, emphasizing the application of psychometric methods, particularly Item Response Theory (IRT), to evaluate the reliability of AI-assisted grading. We examine how grading rubrics can be iteratively refined and how threshold parameters can determine when AI-generated grades are reliable versus when human intervention is necessary. By adjusting thresholds for correctness measures and uncertainty, AI can grade with high precision, significantly reducing grading workloads while maintaining accuracy. Our findings show that AI can achieve a coefficient of determination of $R^2\approx 0.91$ when handling half of the grading load, and $R^2\approx 0.96$ for one-fifth of the load. These results demonstrate AI's potential to assist in grading large-scale assessments, reducing both human effort and associated costs. However, the study underscores the importance of human oversight in cases of uncertainty or complex problem-solving, ensuring the integrity of the grading process.

References in corpus (10)

• Could an Artificial-Intelligence agent pass an introductory physics course?
• A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scores
• Physics task development of prospective physics teachers using ChatGPT
• How understanding large language models can inform the use of ChatGPT in physics education
• Exploring Generative AI assisted feedback writing for students' written responses to a physics conceptual question with prompt engineering and few-shot learning
• How do physics students evaluate artificial intelligence responses on comprehension questions? A study on the perceived scientific accuracy and linguistic quality
• The Impact of AI in Physics Education: A Comprehensive Review from GCSE to University Levels
• Can an AI-tool grade assignments in an introductory physics course?
• Coupled Multiple-response vs. Free-response Conceptual assessment: An Example from upper-division Physics
• Ethel: A Virtual Teaching Assistant

Cited by in corpus (5)

• Multilingual Performance of a Multimodal Artificial Intelligence System on Multisubject Physics Concept Inventories
• The Boiling-Frog Problem of Physics Education
• Using Large Language Models for Solving Thermodynamic Problems
• Can Large Language Models Correctly Interpret Equations with Errors?
• When AI Evaluates Its Own Work: Validating Learner-Initiated, AI-Generated Physics Practice Problems