The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development

Abstract

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model's responses. We developed a prototype system -- the Programmer's Assistant -- in order to explore the utility of conversational interactions grounded in code, as well as software engineers' receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant's capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.

arXiv comment43 pages, 3 figures. To be published in IUI 2023

References in corpus (13)

• Training language models to follow instructions with human feedback
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• LaMDA: Language Models for Dialog Applications
• How should my chatbot interact? A survey on human-chatbot interaction design
• Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
• Improving alignment of dialogue agents via targeted human judgements
• BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks
• Better Together? An Evaluation of AI-Supported Code Translation
• A Test for Evaluating Performance in Human-Computer Systems
• Collaborative Software Development on the Web

Cited by in corpus (32)

• The Metacognitive Demands and Opportunities of Generative AI
• A Survey on Large Language Models for Code Generation
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
• Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education
• Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
• Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Data
• Iris: An AI-Driven Virtual Tutor For Computer Science Education
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verification
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges
• Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat
• Conversate: Supporting Reflective Learning in Interview Practice Through Interactive Simulation and Dialogic Feedback
• Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support
• Validating AI-Generated Code with Live Programming
• CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
• Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach
• Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation
• Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
• To Search or To Gen? Exploring the Synergy between Generative AI and Web Search in Programming
• Anticipating User Needs: Insights from Design Fiction on Conversational Agents for Computational Thinking
• Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
• Revolutionizing Newcomers' Onboarding Process in OSS Communities: The Future AI Mentor
• OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models
• GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication
• A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice
• CoGrader: Transforming Instructors' Assessment of Project Reports through Collaborative LLM Integration
• AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations
• Mitigating the Carbon Footprint of Chatbots as Consumers
• What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issues