On Learning Meaningful Assert Statements for Unit Test Cases

Abstract

Software testing is an essential part of the software lifecycle and requires a substantial amount of time and effort. It has been estimated that software developers spend close to 50% of their time on testing the code they write. For these reasons, a long standing goal within the research community is to (partially) automate software testing. While several techniques and tools have been proposed to automatically generate test methods, recent work has criticized the quality and usefulness of the assert statements they generate. Therefore, we employ a Neural Machine Translation (NMT) based approach called Atlas(AuTomatic Learning of Assert Statements) to automatically generate meaningful assert statements for test methods. Given a test method and a focal method (i.e.,the main method under test), Atlas can predict a meaningful assert statement to assess the correctness of the focal method. We applied Atlas to thousands of test methods from GitHub projects and it was able to predict the exact assert statement manually written by developers in 31% of the cases when only considering the top-1 predicted assert. When considering the top-5 predicted assert statements, Atlas is able to predict exact matches in 50% of the cases. These promising results hint to the potential usefulness ofour approach as (i) a complement to automatic test case generation techniques, and (ii) a code completion support for developers, whocan benefit from the recommended assert statements while writing test code.

Cited by in corpus (18)

• Practical Program Repair in the Era of Large Pre-trained Language Models
• TOGA: A Neural Method for Test Oracle Generation
• An Empirical Study on the Usage of Transformer Models for Code Completion
• Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
• Neural-Based Test Oracle Generation: A Large-scale Evaluation and Lessons Learned
• Perfect is the enemy of test oracle
• TOGLL: Correct and Strong Test Oracle Generation with LLMs
• Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models
• Toward a Theory of Causation for Interpreting Neural Code Models
• Assessing Evaluation Metrics for Neural Test Oracle Generation
• Learning the Relation between Code Features and Code Transforms with Structured Prediction
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Mokav: Execution-driven Differential Testing with LLMs
• AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model
• SAGA: Summarization-Guided Assert Statement Generation
• Which Syntactic Capabilities Are Statistically Learned by Masked Language Models for Code?
• Automatically Detecting Numerical Instability in Machine Learning Applications via Soft Assertions
• Evaluating LLM-Based Regression Test Generation