Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Abstract

The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate. An emerging body of work has looked into using LLMs in education, but few have examined the impacts of LLMs on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the control group). Within the experimental group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability, though they also had concerns about CodeTutor's limited role in developing critical thinking skills. Over the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon our results, we discuss the implications of our findings for integrating Generative AI literacy into curricula to foster critical thinking skills and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

arXiv commentAccepted to Learning @ Scale 2024

References in corpus (18)

• Emergent Abilities of Large Language Models
• Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
• The Robots are Here: Navigating the Generative AI Revolution in Computing Education
• BloombergGPT: A Large Language Model for Finance
• Large Language Models Are Human-Level Prompt Engineers
• ChatGPT and Software Testing Education: Promises & Perils
• Using Large Language Models to Enhance Programming Error Messages
• Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
• TrustLLM: Trustworthiness in Large Language Models
• AI-assisted coding: Experiments with GPT-4
• A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
• Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models
• Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors
• "The teachers are confused as well": A Multiple-Stakeholder Ethics Discussion on Large Language Models in Computing Education
• AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs
• Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course?
• Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code
• Experiences with Remote Examination Formats in Light of GPT-4

Cited by in corpus (6)

• Towards responsible AI for education: Hybrid human-AI to confront the Elephant in the room
• Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting
• Utilizing ChatGPT in a Data Structures and Algorithms Course: A Teaching Assistant's Perspective
• That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai
• JELAI: Integrating AI and Learning Analytics in Jupyter Notebooks
• Observing Without Doing: Pseudo-Apprenticeship Patterns in Student LLM Use