Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation

Abstract

Research shows that errors in natural language can be corrected by translating texts to another language and back using language models. We explore to what extent this latent correction capability extends to Automated Program Repair (APR) by investigating Round-Trip Translation (RTT): translating code from one programming language into another programming or natural language and back, using Large Language Models (LLMs). We hypothesize that RTT restores patterns most commonly seen in the LLM's training corpora through regression toward the mean, replacing infrequent bugs with more frequent, natural, bug-free code. To test this hypothesis, we employ nine LLMs and four common APR benchmarks in Java, and perform a detailed quantitative and qualitative analysis of RTT-generated patches. We find that RTT through English generates plausible patches for 100 of 164 bugs with GPT-4 on the HumanEval-Java benchmark, and 97 are found to be correct in our manual assessment. Moreover, RTT uniquely generates plausible patches for 46 bugs that were missed by LLMs specifically fine-tuned for APR. While this demonstrates the viability of RTT for APR, we also observe limitations, such as a lower overall bug fix rate than the state-of-the-art and diluting the original coding style. We analyze the impact of these limitations and discuss the potential of using RTT as a complementary component in APR frameworks. A replication package is available for download from https://doi.org/10.5281/zenodo.10500593. Keywords: automated program repair, large language model, machine translation

Accepted for publication in ACM Transactions on Software Engineering and Methodology (TOSEM)

References in corpus (28)

• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• Scaling Laws for Neural Language Models
• Emergent Abilities of Large Language Models
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• Practical Program Repair in the Era of Large Pre-trained Language Models
• Automatic Software Repair: a Bibliography
• Reflexion: Language Agents with Verbal Reinforcement Learning
• CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
• Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
• An Empirical Study of the Non-determinism of ChatGPT in Code Generation
• GPT-4o System Card
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• Dissection of a Bug Dataset: Anatomy of 395 Patches from Defects4J
• Teaching Large Language Models to Self-Debug
• A Comprehensive Study of Automatic Program Repair on the QuixBugs Benchmark
• Exploring Software Naturalness through Neural Language Models
• Fully Autonomous Programming with Large Language Models
• CM3: A Causal Masked Multimodal Model of the Internet
• Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code
• Conversational Automated Program Repair
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation
• Efficient Large Scale Language Modeling with Mixtures of Experts
• Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation
• Human-AI Coevolution
• De-Hallucinator: Mitigating LLM Hallucinations in Code Generation Tasks via Iterative Grounding
• Fully Autonomous Programming using Iterative Multi-Agent Debugging with Large Language Models
• A Survey of Learning-based Automated Program Repair
• MUFIN: Improving Neural Repair Models with Back-Translation