A Comprehensive Survey of Benchmarks for Automated Improvement of Software's Non-Functional Properties

Abstract

Performance is a key quality of modern software. Although recent years have seen a spike in research on automated improvement of software's execution time, energy, memory consumption, etc., there is a noticeable lack of standard benchmarks for such work. It is also unclear how such benchmarks are representative of current software. Furthermore, frequently non-functional properties of software are targeted for improvement one-at-a-time, neglecting potential negative impact on other properties. In order to facilitate more research on automated improvement of non-functional properties of software, we conducted a survey gathering benchmarks used in previous work. We considered 5 major online repositories of software engineering work: ACM Digital Library, IEEE Xplore, Scopus, Google Scholar, and ArXiV. We gathered 5000 publications (3749 unique), which were systematically reviewed to identify work that empirically improves non-functional properties of software. We identified 386 relevant papers. We find that execution time is the most frequently targeted property for improvement (in 62% of relevant papers), while multi-objective improvement is rarely considered (5%). Static approaches are prevalent (in 53% of papers), with exploratory approaches (evolutionary in 18% and non-evolutionary in 14% of papers) increasingly popular in the last 10 years. Only 40% of 386 papers describe work that uses benchmark suites, rather than single software, of those SPEC is most popular (covered in 33 papers). We also provide recommendations for choice of benchmarks in future work, noting, e.g., lack of work that covers Python or JavaScript. We provide all programs found in the 386 papers on our dedicated webpage at https://bloa.github.io/nfunc_survey/ We hope that this effort will facilitate more research on the topic of automated improvement of software's non-functional properties.

43 pages, 18 figures

References in corpus (10)

• A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem
• Factors Influencing Quality of Mobile Apps:Role of Mobile App Development Life Cycle
• Adapting Multi-objectivized Software Configuration Tuning
• Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
• Trimming Mobile Applications for Bandwidth-Challenged Networks in Developing Regions
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• Binary Debloating for Security via Demand Driven Loading
• GEVO: GPU Code Optimization using Evolutionary Computation
• NEAT: A Framework for Automated Exploration of Floating Point Approximations
• A Variable Vector Length SIMD Architecture for HW/SW Co-designed Processors