Validating AI-Generated Code with Live Programming

Abstract

AI-powered programming assistants are increasingly gaining popularity, with GitHub Copilot alone used by over a million developers worldwide. These tools are far from perfect, however, producing code suggestions that may be incorrect in subtle ways. As a result, developers face a new challenge: validating AI's suggestions. This paper explores whether Live Programming (LP), a continuous display of a program's runtime values, can help address this challenge. To answer this question, we built a Python editor that combines an AI-powered programming assistant with an existing LP environment. Using this environment in a between-subjects study (N=17), we found that by lowering the cost of validation by execution, LP can mitigate over- and under-reliance on AI-generated programs and reduce the cognitive load of validation for certain types of tasks.

arXiv comment8 pages, 4 figures

References in corpus (6)

• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• Perfection Not Required? Human-AI Partnerships in Code Translation
• Interactive Code Generation via Test-Driven User-Intent Formalization
• Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
• A Large-Scale Survey on the Usability of AI Programming Assistants: Successes and Challenges
• LExecutor: Learning-Guided Execution

Cited by in corpus (4)

• VideoDiff: Human-AI Video Co-Creation with Alternatives
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• PLAID: Supporting Computing Instructors to Identify Domain-Specific Programming Plans at Scale