TOGA: A Neural Method for Test Oracle Generation

Abstract

Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality. In this paper, we propose TOGA (a neural method for Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method. Our approach can handle units with ambiguous or missing documentation, and even units with a missing implementation. We evaluate our approach on both oracle inference accuracy and functional bug-finding. Our technique improves accuracy by 33\% over existing oracle inference approaches, achieving 96\% overall accuracy on a held out test dataset. Furthermore, we show that when integrated with a automated test generation tool (EvoSuite), our approach finds 57 real world bugs in large-scale Java programs, including 30 bugs that are not found by any other automated testing method in our evaluation.

arXiv commentTo appear in ICSE 2022

References in corpus (5)

• Learning and Evaluating Contextual Embedding of Source Code
• On Learning Meaningful Assert Statements for Unit Test Cases
• Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformers
• Automated Unit Test Generation for Python
• Program Merge Conflict Resolution via Neural Transformers

Cited by in corpus (11)

• Practical Program Repair in the Era of Large Pre-trained Language Models
• Automating the Correctness Assessment of AI-generated Code for Security Contexts
• Neural-Based Test Oracle Generation: A Large-scale Evaluation and Lessons Learned
• Assessing Evaluation Metrics for Neural Test Oracle Generation
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• AsserT5: Test Assertion Generation Using a Fine-Tuned Code Language Model
• Mokav: Execution-driven Differential Testing with LLMs
• SAGA: Summarization-Guided Assert Statement Generation
• Morescient GAI for Software Engineering (Extended Version)
• Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation
• LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review