PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation

Abstract

Test-driven development (TDD) is a widely-employed software development practice that mandates writing test cases based on requirements before writing the actual code. While writing test cases is the centerpiece of TDD, it is time-consuming, expensive, and often shunned by developers. To address these issues associated with TDD, automated test case generation approaches have recently been investigated. Such approaches take source code as input, but not the requirements. Therefore, existing work does not fully support true TDD, as actual code is required to generate test cases. In addition, current deep learning-based test case generation approaches are trained with one learning objective, i.e., to generate test cases that are exactly matched with the ground-truth test cases. However, such approaches may limit the model's ability to generate different yet correct test cases. In this paper, we introduce PyTester, a Text-to-Testcase generation approach that can automatically generate syntactically correct, executable, complete, and effective test cases while being aligned with a given natural language requirement. We evaluate PyTester on the public APPS benchmark dataset, and the results show that our Deep RL approach enables PyTester, a small language model, to outperform much larger language models like GPT3.5, StarCoder, and InCoder. Our findings suggest that future research could consider improving small over large LMs for better resource efficiency by integrating the SE domain knowledge into the design of reinforcement learning architecture.

17 pages, 5 figures

References in corpus (16)

• Training language models to follow instructions with human feedback
• Evaluating Large Language Models Trained on Code
• Code Llama: Open Foundation Models for Code
• Towards Out-Of-Distribution Generalization: A Survey
• StarCoder: may the source be with you!
• Measuring Coding Challenge Competence With APPS
• InCoder: A Generative Model for Code Infilling and Synthesis
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• TOGA: A Neural Method for Test Oracle Generation
• On Learning Meaningful Assert Statements for Unit Test Cases
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• CodeT: Code Generation with Generated Tests
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
• ChatUniTest: A Framework for LLM-Based Test Generation
• Hyperparameter Tuning for Deep Reinforcement Learning Applications

Cited by in corpus (1)

• Mokav: Execution-driven Differential Testing with LLMs