"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers

Abstract

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

26 pages, 2 figures, TOCHI

References in corpus (10)

• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Automatic Generation of Programming Exercises and Code Explanations using Large Language Models
• Using Large Language Models to Enhance Programming Error Messages
• Comparing Code Explanations Created by Students and Large Language Models
• A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level
• Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?
• Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?
• AI-driven Development Is Here: Should You Worry?
• A Survey of Automated Programming Hint Generation -- The HINTS Framework
• The Role of Working Memory in Program Tracing

Cited by in corpus (22)

• The Robots are Here: Navigating the Generative AI Revolution in Computing Education
• CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
• Desirable Characteristics for AI Teaching Assistants in Programming Education
• Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• Could ChatGPT get an Engineering Degree? Evaluating Higher Education Vulnerability to AI Assistants
• Insights from Social Shaping Theory: The Appropriation of Large Language Models in an Undergraduate Programming Course
• What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
• Beyond Code Generation: LLM-supported Exploration of the Program Design Space
• How Do Programming Students Use Generative AI?
• Enhancing Programming Error Messages in Real Time with Generative AI
• The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
• Understanding Human-AI Trust in Education
• Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
• How Scientists Use Large Language Models to Program
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
• Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
• Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
• GenAI Voice Mode in Programming Education
• Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
• Understanding User Mental Models in AI-Driven Code Completion Tools: Insights from an Elicitation Study
• EditFlow: Benchmarking and Optimizing Code Edit Recommendation Systems via Reconstruction of Developer Flows