AI-enhanced Auto-correction of Programming Exercises: How Effective is GPT-3.5?

Abstract

Timely formative feedback is considered as one of the most important drivers for effective learning. Delivering timely and individualized feedback is particularly challenging in large classes in higher education. Recently Large Language Models such as GPT-3 became available to the public that showed promising results on various tasks such as code generation and code explanation. This paper investigates the potential of AI in providing personalized code correction and generating feedback. Based on existing student submissions of two different real-world assignments, the correctness of the AI-aided e-assessment as well as the characteristics such as fault localization, correctness of hints, and code style suggestions of the generated feedback are investigated. The results show that 73 % of the submissions were correctly identified as either correct or incorrect. In 59 % of these cases, GPT-3.5 also successfully generated effective and high-quality feedback. Additionally, GPT-3.5 exhibited weaknesses in its evaluation, including localization of errors that were not the actual errors, or even hallucinated errors. Implications and potential new usage scenarios are discussed.

arXiv commentaccepted to the International Journal of Engineering Pedagogy (iJEP; eISSN: 2192-4880)

References in corpus (10)

• Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
• Using Large Language Models to Enhance Programming Error Messages
• Comparing Code Explanations Created by Students and Large Language Models
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
• Is ChatGPT the Ultimate Programming Assistant -- How far is it?
• Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models
• Can ChatGPT Pass An Introductory Level Functional Language Programming Course?
• Computing Education in the Era of Generative AI
• Exploring the Potential of Large Language Models to Generate Formative Programming Feedback

Cited by in corpus (4)

• Feedback-Generation for Programming Exercises With GPT-4
• Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
• Automating Autograding: Large Language Models as Test Suite Generators for Introductory Programming
• A systematic comparison of Large Language Models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor