Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code

Abstract

Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation -- with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset -- consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs -- can help drive research in this area.

arXiv commentPublished in ICSE 2024

References in corpus (5)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
• Perfection Not Required? Human-AI Partnerships in Code Translation
• Mono2Micro: A Practical and Effective Tool for Decomposing Monolithic Java Applications to Microservices
• Better Together? An Evaluation of AI-Supported Code Translation

Cited by in corpus (13)

• Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Models
• AlphaTrans: A Neuro-Symbolic Compositional Approach for Repository-Level Code Translation and Validation
• Type-Constrained Code Generation with Language Models
• One Size Does Not Fit All: Investigating Efficacy of Perplexity in Detecting LLM-Generated Code
• ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs
• Automated Description Generation for Software Patches
• Model-Driven Quantum Code Generation Using Large Language Models and Retrieval-Augmented Generation
• IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
• A Preliminary Study of Multilingual Code Language Models for Code Generation Task Using Translated Benchmarks
• Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation
• Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models
• Can Emulating Semantic Translation Help LLMs with Code Translation? A Study Based on Pseudocode
• Program Skeletons for Automated Program Translation