Metamorphic Testing: A New Approach for Generating Next Test Cases
arXiv:2002.12543
Abstract
In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.
11 pages, Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology
Cited by in corpus (34)
- Machine Learning Testing: Survey, Landscapes and Horizons
- How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
- A Systematic Impact Study for Fuzzer-Found Compiler Bugs
- SemMT: A Semantic-based Testing Approach for Machine Translation Systems
- Testing Database Engines via Pivoted Query Synthesis
- GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming
- Metamorphic Testing for Object Detection Systems
- Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations
- Metamorphic Detection of Adversarial Examples in Deep Learning Models With Affine Transformations
- MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases
- Challenges for Verifying and Validating Scientific Software in Computational Materials Science
- A Replication Study on Predicting Metamorphic Relations at Unit Testing Level
- Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications
- Gauntlet: Finding Bugs in Compilers for Programmable Packet Processing
- On the Potential and Limitations of Few-Shot In-Context Learning to Generate Metamorphic Specifications for Tax Preparation Software
- Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relations
- DarthShader: Fuzzing WebGPU Shader Translators & Compilers
- Intramorphic Testing: A New Approach to the Test Oracle Problem
- Automatic Testing and Improvement of Machine Translation
- Improving The Effectiveness of Automatically Generated Test Suites Using Metamorphic Testing
- Object-based Metamorphic Testing through Image Structuring
- Coupled Requirements-driven Testing of CPS: From Simulation To Reality
- Metamorphic Testing of Multimodal Human Trajectory Prediction
- Structure-Invariant Testing for Machine Translation
- Towards Structured Evaluation of Deep Neural Network Supervisors
- Validation of Automatically Generated Patches: An Appetizer
- Towards a Complete Metamorphic Testing Pipeline
- MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
- HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks
- HornFuzz: Fuzzing CHC solvers
- Metamorphic Testing for Quality Assurance of Protein Function Prediction Tools
- Testing DBMS Performance with Mutations
- Using Metamorphic Relations to Verify and Enhance Artcode Classification
- Testing Machine Translation via Referential Transparency