Benchmarking as Empirical Standard in Software Engineering Research

Abstract

In empirical software engineering, benchmarks can be used for comparing different methods, techniques and tools. However, the recent ACM SIGSOFT Empirical Standards for Software Engineering Research do not include an explicit checklist for benchmarking. In this paper, we discuss benchmarks for software performance and scalability evaluation as example research areas in software engineering, relate benchmarks to some other empirical research methods, and discuss the requirements on benchmarks that may constitute the basis for a checklist of a benchmarking standard for empirical software engineering research.

This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in EASE 2021 (Evaluation and Assessment in Software Engineering), June 21-23, 2021, Trondheim, Norway, https://doi.org/10.1145/3463274.3463361

References in corpus (1)

• SPRAT: A Spatially-Explicit Marine Ecosystem Model Based on Population Balance Equations

Cited by in corpus (6)

• ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks
• Are Prompts All You Need? Evaluating Prompt-Based Large Language Models (LLM)s for Software Requirements Classification
• Towards Solving the Challenge of Minimal Overhead Monitoring
• When Should I Run My Application Benchmark?: Studying Cloud Performance Variability for the Case of Stream Processing Applications
• Innamark: A Whitespace Replacement Information-Hiding Method
• Using Data Redundancy Techniques to Detect and Correct Errors in Logical Data