CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
arXiv:2401.11314 · doi:10.1145/3613904.3642773
Abstract
Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI's unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.
CHI 2024 Paper - The paper includes 17 pages, 8 figures, 2 tables, along with a 2-page appendix
References in corpus (7)
- On the Opportunities and Risks of Foundation Models
- Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
- Using Large Language Models to Enhance Programming Error Messages
- "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
- Comparing Code Explanations Created by Students and Large Language Models
- Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Cited by in corpus (19)
- Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
- Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course
- Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
- 61A Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff. (Now What?)
- Are We On Track? AI-Assisted Active and Passive Goal Reflection During Meetings
- One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
- Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
- From Pilots to Practices: A Scoping Review of GenAI-Enabled Personalization in Computer Science Education
- Evaluating adaptive and generative AI-based feedback and recommendations in a knowledge-graph-integrated programming learning system
- That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai
- GenAI Voice Mode in Programming Education
- A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice
- JELAI: Integrating AI and Learning Analytics in Jupyter Notebooks
- PLAID: Supporting Computing Instructors to Identify Domain-Specific Programming Plans at Scale
- Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
- Designing for Novice Debuggers: A Pilot Study on an AI-Assisted Debugging Tool
- Simulated Interactive Debugging
- A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
- Self-Regulated Personal Contracts as a Harm Reduction Approach to Generative AI in Undergraduate Programming Education