Evaluating Search-Based Software Microbenchmark Prioritization
arXiv:2211.13525 · doi:10.1109/TSE.2024.3380836
Abstract
Ensuring that software performance does not degrade after a code change is paramount. A solution is to regularly execute software microbenchmarks, a performance testing technique similar to (functional) unit tests, which, however, often becomes infeasible due to extensive runtimes. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, drastically reducing their potential application. In this paper, we empirically evaluate single- and multi-objective search-based microbenchmark prioritization techniques to understand whether they are more effective and efficient than greedy, coverage-based techniques. For this, we devise three search objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that search algorithms (SAs) are only competitive with but do not outperform the best greedy, coverage-based baselines. However, a simple greedy technique utilizing solely the performance change history (without coverage information) is equally or more effective than the best coverage-based techniques while being considerably more efficient, with a runtime overhead of less than 1%. These results show that simple, non-coverage-based techniques are a better fit for microbenchmarks than complex coverage-based techniques.
17 pages, 7 figures, 4 tables, 1 listing; accepted in IEEE Transactions on Software Engineering
References in corpus (10)
- A Large-Scale Empirical Comparison of Static and Dynamic Test Case Prioritization Techniques
- Test Prioritization in Continuous Integration Environments
- How Do Static and Dynamic Test Case Prioritization Techniques Perform on Modern Software Systems? An Extensive Study on GitHub Projects
- Change Point Detection in Software Performance Testing
- Towards effective assessment of steady state performance in Java software: Are we there yet?
- GLIB: Towards Automated Test Oracle for Graphically-Rich Applications
- Quantifying Performance Changes with Effect Size Confidence Intervals
- Creating a Virtuous Cycle in Performance Testing at MongoDB
- Evaluating Search-Based Software Microbenchmark Prioritization
- Using Microbenchmark Suites to Detect Application Performance Changes