An Empirical Study of the Non-determinism of ChatGPT in Code Generation

Abstract

There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.

References in corpus (4)

• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• ChatGPT and Software Testing Education: Promises & Perils
• COMET: Coverage-guided Model Generation For Deep Learning Library Testing
• Fault-Aware Neural Code Rankers

Cited by in corpus (16)

• The Metacognitive Demands and Opportunities of Generative AI
• What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
• Will Your Next Pair Programming Partner Be Human? An Empirical Evaluation of Generative AI as a Collaborative Teammate in a Semester-Long Classroom Setting
• Large Language Models for Automated Web-Form-Test Generation: An Empirical Study
• Statically Contextualizing Large Language Models with Typed Holes
• Search-based Optimisation of LLM Learning Shots for Story Point Estimation
• Piloting Copilot, Codex, and StarCoder2: Hot Temperature, Cold Prompts, or Black Magic?
• Cognitive phantoms in LLMs through the lens of latent variables
• Using language models to label clusters of scientific documents
• EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices
• Harnessing Language for Coordination: A Framework and Benchmark for LLM-Driven Multi-Agent Control
• TaskEval: Assessing Difficulty of Code Generation Tasks for Large Language Models
• On Developers' Self-Declaration of AI-Generated Code: An Analysis of Practices
• Architectures of Error: A Philosophical Inquiry into AI and Human Code Generation
• Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation
• Ethical AI prompt recommendations in large language models using collaborative filtering