IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics

Abstract

IOHprofiler is a new tool for analyzing and comparing iterative optimization heuristics. Given as input algorithms and problems written in C or Python, it provides as output a statistical evaluation of the algorithms' performance by means of the distribution on the fixed-target running time and the fixed-budget function values. In addition, IOHprofiler also allows to track the evolution of algorithm parameters, making our tool particularly useful for the analysis, comparison, and design of (self-)adaptive algorithms. IOHprofiler is a ready-to-use software. It consists of two parts: an experimental part, which generates the running time data, and a post-processing part, which produces the summarizing comparisons and statistical evaluations. The experimental part is build on the COCO software, which has been adjusted to cope with optimization problems that are formulated as functions $f:{\mathcal{S}}^n\to \mathbb{R}$ with $\mathcal{S}$ being a discrete alphabet of integers. The post-processing part is our own work. It can be used as a stand-alone tool for the evaluation of running time data of arbitrary benchmark problems. It accepts as input files not only the output files of IOHprofiler, but also original COCO data files. The post-processing tool is designed for an interactive evaluation, allowing the user to chose the ranges and the precision of the displayed data according to his/her needs. IOHprofiler is available on GitHub at \url{https://github.com/IOHprofiler}.

Cited by in corpus (16)

• Benchmarking in Optimization: Best Practice and Open Issues
• Benchmarking Discrete Optimization Heuristics with IOHprofiler
• The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection
• Automated Configuration of Genetic Algorithms by Tuning for Anytime Performance
• EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions
• Benchmarking a $(\mu +\lambda )$ Genetic Algorithm with Configurable Crossover Probability
• Fast Mutation in Crossover-based Algorithms
• Fixed-Target Runtime Analysis
• A Performance Analysis of Basin Hopping Compared to Established Metaheuristics for Global Optimization
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
• Meta Learning Black-Box Population-Based Optimizers
• Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB
• Extensible Logging and Empirical Attainment Function for IOHexperimenter
• Sequential vs. Integrated Algorithm Selection and Configuration: A Case Study for the Modular CMA-ES
• Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection