Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions

Abstract

Recent research has explored the creation of questions from code submitted by students. These Questions about Learners' Code (QLCs) are created through program analysis, exploring execution paths, and then creating code comprehension questions from these paths and the broader code structure. Responding to the questions requires reading and tracing the code, which is known to support students' learning. At the same time, computing education researchers have witnessed the emergence of Large Language Models (LLMs) that have taken the community by storm. Researchers have demonstrated the applicability of these models especially in the introductory programming context, outlining their performance in solving introductory programming problems and their utility in creating new learning resources. In this work, we explore the capability of the state-of-the-art LLMs (GPT-3.5 and GPT-4) in answering QLCs that are generated from code that the LLMs have created. Our results show that although the state-of-the-art LLMs can create programs and trace program execution when prompted, they easily succumb to similar errors that have previously been recorded for novice programmers. These results demonstrate the fallibility of these models and perhaps dampen the expectations fueled by the recent LLM hype. At the same time, we also highlight future research possibilities such as using LLMs to mimic students as their behavior can indeed be similar for some specific tasks.

References in corpus (14)

• Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• Using Large Language Models to Enhance Programming Error Messages
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• Comparing Code Explanations Created by Students and Large Language Models
• Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
• Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
• Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?
• Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
• Learnersourcing in the Age of AI: Student, Educator and Machine Partnerships for Content Creation
• Automatically Generating CS Learning Materials with Large Language Models
• Students Struggle to Explain Their Own Program Code
• Let's Ask Students About Their Programs, Automatically
• Automated Questions About Learners' Own Code Help to Detect Fragile Knowledge

Cited by in corpus (1)

• A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice