Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
arXiv:2509.22974 · doi:10.1145/3769994.3770034
Abstract
How students utilize immediate tutoring feedback in programming education depends on various factors. Among them are the feedback quality, but also students' engagement, i.e., their perception, interpretation, and use of feedback. However, there is limited research on how students engage with various types of tutoring feedback. For this reason, we developed a learning environment that provides students with Python programming tasks and various types of immediate, AI-generated tutoring feedback. The feedback is displayed within four components. Using a mixed-methods approach (think-aloud study and eye-tracking), we conducted a study with 20 undergraduate students enrolled in an introductory programming course. Our research aims to: (1) identify what students think when they engage with the tutoring feedback components, and (2) explore the relations between the tutoring feedback components, students' visual attention, verbalized thoughts, and their immediate actions as part of the problem-solving process. The analysis of students' thoughts while engaging with 380 feedback components revealed four main themes: students express understanding or disagreement, additional information needed, and students explicitly judge the feedback. Exploring the relations between feedback, students' attention, thoughts, and actions showed a clear relationship. While expressions of understanding were associated with improvements, expressions of disagreement or need for additional information prompted students to collect another feedback component rather than act on the current information. These insights into students' engagement and decision-making processes contribute to an increased understanding of tutoring feedback and how students engage with it. Thereby, this work has implications for tool developers and educators facilitating feedback.
Accepted for the 25th International Conference on Computing Education Research (Koli Calling '25)
References in corpus (16)
- Robust Speech Recognition via Large-Scale Weak Supervision
- The Robots are Here: Navigating the Generative AI Revolution in Computing Education
- CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
- Automated Grading and Feedback Tools for Programming Education: A Systematic Review
- Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
- Feedback-Generation for Programming Exercises With GPT-4
- Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
- Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
- One Step at a Time: Combining LLMs and Static Analysis to Generate Next-Step Hints for Programming Tasks
- Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation
- How Helpful do Novice Programmers Find the Feedback of an Automated Repair Tool?
- That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai
- Evaluating Language Models for Generating and Judging Programming Feedback
- Unlimited Practice Opportunities: Automated Generation of Comprehensive, Personalized Programming Tasks
- GenAI Voice Mode in Programming Education
- Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?