Better Together? An Evaluation of AI-Supported Code Translation

Abstract

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.

arXiv comment35 pages, 3 figures. To be published in IUI 2022

References in corpus (12)

• Evaluating Large Language Models Trained on Code
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Expanding Explainability: Towards Social Transparency in AI systems
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
• Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
• Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators
• Increasing the Speed and Accuracy of Data LabelingThrough an AI Assisted Interface
• Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
• Designing for Democratization: Introducing Novices to Artificial Intelligence Via Maker Kits
• Guru, Partner, or Pencil Sharpener? Understanding Designers' Attitudes Towards Intelligent Creativity Support Tools

Cited by in corpus (10)

• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• The Metacognitive Demands and Opportunities of Generative AI
• Design Principles for Generative AI Applications
• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
• Investigating and Designing for Trust in AI-powered Code Generation Tools
• Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature
• Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
• How Scientists Use Large Language Models to Program
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review