Perfection Not Required? Human-AI Partnerships in Code Translation

Abstract

Generative models have become adept at producing artifacts such as images, videos, and prose at human-like levels of proficiency. New generative techniques, such as unsupervised neural machine translation (NMT), have recently been applied to the task of generating source code, translating it from one programming language to another. The artifacts produced in this way may contain imperfections, such as compilation or logical errors. We examine the extent to which software engineers would tolerate such imperfections and explore ways to aid the detection and correction of those errors. Using a design scenario approach, we interviewed 11 software engineers to understand their reactions to the use of an NMT model in the context of application modernization, focusing on the task of translating source code from one language to another. Our three-stage scenario sparked discussions about the utility and desirability of working with an imperfect AI system, how acceptance of that system's outputs would be established, and future opportunities for generative AI in application modernization. Our study highlights how UI features such as confidence highlighting and alternate translations help software engineers work with and better understand generative NMT models.

arXiv comment18 pages, 1 figure. To be published in IUI 2021

References in corpus (5)

• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities
• Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators
• Exploring AI Futures Through Role Play

Cited by in corpus (17)

• The Metacognitive Demands and Opportunities of Generative AI
• Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works
• Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• How Knowledge Workers Think Generative AI Will (Not) Transform Their Industries
• "It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models
• Investigating and Designing for Trust in AI-powered Code Generation Tools
• "It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study
• Validating AI-Generated Code with Live Programming
• Neural language models for network configuration: Opportunities and reality check
• Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
• Output Format Biases in the Evaluation of Large Language Models for Code Translation
• How Scientists Use Large Language Models to Program
• Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations
• A User-Centered Investigation of Personal Music Tours
• Understanding Collaboration between Professional Designers and Decision-making AI: A Case Study in the Workplace
• Program Skeletons for Automated Program Translation