Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
arXiv:2503.06195 · doi:10.1007/s10664-025-10793-0
Abstract
The integration of Artificial Intelligence (AI) into Integrated Development Environments (IDEs) is reshaping software development, fundamentally altering how developers interact with their tools. This shift marks the emergence of Human-AI Experience in Integrated Development Environment (in-IDE HAX), a field that explores the evolving dynamics of Human-Computer Interaction in AI-assisted coding environments. Despite rapid adoption, research on in-IDE HAX remains fragmented, which highlights the need for a unified overview of current practices, challenges, and opportunities. To provide a structured overview of existing research, we conduct a systematic literature review of 90 studies, summarizing current findings and outlining areas for further investigation. We organize key insights from reviewed studies into three aspects: Impact, Design, and Quality of AI-based systems inside IDEs. Impact findings show that AI-assisted coding enhances developer productivity but also introduces challenges, such as verification overhead and over-reliance. Design studies show that effective interfaces surface context, provide explanations and transparency of suggestion, and support user control. Quality studies document risks in correctness, maintainability, and security. For future research, priorities include productivity studies, design of assistance, and audit of AI-generated code. The agenda calls for larger and longer evaluations, stronger audit and verification assets, broader coverage across the software life cycle, and adaptive assistance under user control.
Accepted to Empirical Software Engineering (EMSE) special issue Human-Centered AI for Software Engineering (HumanAISE), 37 pages, 7 figure
References in corpus (22)
- Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
- The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
- CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
- "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
- Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks
- Investigating and Designing for Trust in AI-powered Code Generation Tools
- Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
- Better Together? An Evaluation of AI-Supported Code Translation
- Practices and Challenges of Using GitHub Copilot: An Empirical Study
- How Readable is Model-generated Code? Examining Readability and Visual Inspection of GitHub Copilot
- Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
- Ivie: Lightweight Anchored Explanations of Just-Generated Code
- Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concerns
- Validating AI-Generated Code with Live Programming
- CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
- Anticipating User Needs: Insights from Design Fiction on Conversational Agents for Computational Thinking
- Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysis
- A User-centered Security Evaluation of Copilot
- Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot
- A Transformer-Based Approach for Smart Invocation of Automatic Code Completion
- Can Developers Prompt? A Controlled Experiment for Code Documentation Generation
- Towards Feature Engineering with Human and AI's Knowledge: Understanding Data Science Practitioners' Perceptions in Human&AI-Assisted Feature Engineering Design