LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review

Abstract

Large language models (LLMs) increasingly decide whether software behaves correctly, either by writing a test oracle or by acting as one. Yet two oracles can look identical and rest on different ground: one assertion encodes a written specification, another only what the model learned in training. Prior secondary studies sort oracles by form or by technique, rarely by the property that governs how far a verdict can be trusted: where its authority comes from. This systematic literature review, reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, screens 2,436 records to 54 included studies, extended by citation searching (snowballing) to 83 in total. We read the corpus along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. Just over half of the corpus reaches a verdict with no specification at all. That is what lets these oracles work on code with no specification to consult, and what leaves a challenged verdict with less to fall back on. Source and mechanism cross-cut rather than coincide, so a label such as LLM-as-a-judge names how a verdict is produced, not why it should be trusted. Oracle quality is most often judged by resemblance to a known oracle rather than by whether injected faults are caught. The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict. The protocol, search query, and per-study coding sheet are released.

21 pages, 10 figures, 11 tables. Systematic literature review of 83 studies, reported under PRISMA 2020. Published in IEEE Access. Replication package: https://doi.org/10.5281/zenodo.21194940

References in corpus (13)

• TOGA: A Neural Method for Test Oracle Generation
• Neural-Based Test Oracle Generation: A Large-scale Evaluation and Lessons Learned
• TOGLL: Correct and Strong Test Oracle Generation with LLMs
• Assessing Evaluation Metrics for Neural Test Oracle Generation
• Acceptance Test Generation with Large Language Models: An Industrial Case Study
• AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions
• SAGA: Summarization-Guided Assert Statement Generation
• Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation
• Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models
• LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review
• MR-Coupler: Automated Metamorphic Test Generation via Functional Coupling Analysis
• CASCADE: Detecting Inconsistencies between Code and Documentation with Automatic Test Generation
• POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference

Cited by in corpus (1)

• LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review