GenAI Voice Mode in Programming Education
arXiv:2509.10596 · doi:10.1145/3769994.3770001
Abstract
Real-time voice interfaces using multimodal Generative AI (GenAI) can potentially address the accessibility needs of novice programmers with disabilities (e.g., related to vision). Yet, little is known about how novices interact with GenAI tools and their feedback quality in the form of audio output. This paper analyzes audio dialogues from nine 9th-grade students using a voice-enabled tutor (powered by OpenAI's Realtime API) in an authentic classroom setting while learning Python. We examined the students' voice prompts and AI's responses (1210 messages) by using qualitative coding. We also gathered students' perceptions via the Partner Modeling Questionnaire. The GenAI Voice Tutor primarily offered feedback on mistakes and next steps, but its correctness was limited (71.4% correct out of 416 feedback outputs). Quality issues were observed, particularly when the AI attempted to utter programming code elements. Students used the GenAI voice tutor primarily for debugging. They perceived it as competent, only somewhat human-like, and flexible. The present study is the first to explore the interaction dynamics of real-time voice GenAI tutors and novice programmers, informing future educational tool design and potentially addressing accessibility needs of diverse learners.
Accepted for the 25th International Conference on Computing Education Research (Koli Calling '25)
References in corpus (13)
- Robust Speech Recognition via Large-Scale Weak Supervision
- The Robots are Here: Navigating the Generative AI Revolution in Computing Education
- CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
- "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
- GPT-4o System Card
- User Interaction Patterns and Breakdowns in Conversing with LLM-Powered Voice Assistants
- Evaluating the Application of Large Language Models to Generate Feedback in Programming Education
- Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation
- The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners
- That's Not the Feedback I Need! -- Student Engagement with GenAI Feedback in the Tutor Kai
- Unlimited Practice Opportunities: Automated Generation of Comprehensive, Personalized Programming Tasks
- Students' Feedback Requests and Interactions with the SCRIPT Chatbot: Do They Get What They Ask For?
- Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study