Benchmarking Discrete Optimization Heuristics with IOHprofiler

Abstract

Automated benchmarking environments aim to support researchers in understanding how different algorithms perform on different types of optimization problems. Such comparisons provide insights into the strengths and weaknesses of different approaches, which can be leveraged into designing new algorithms and into the automation of algorithm selection and configuration. With the ultimate goal to create a meaningful benchmark set for iterative optimization heuristics, we have recently released IOHprofiler, a software built to create detailed performance comparisons between iterative optimization heuristics. With this present work we demonstrate that IOHprofiler provides a suitable environment for automated benchmarking. We compile and assess a selection of 23 discrete optimization problems that subscribe to different types of fitness landscapes. For each selected problem we compare performances of twelve different heuristics, which are as of now available as baseline algorithms in IOHprofiler. We also provide a new module for IOHprofiler which extents the fixed-target and fixed-budget results for the individual problems by ECDF results, which allows one to derive aggregated performance statistics for groups of problems.

References in corpus (10)

• Ising formulations of many NP problems
• Better Runtime Guarantees Via Stochastic Domination
• The (1+$\lambda$) Evolutionary Algorithm with Self-Adjusting Mutation Rate
• IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics
• Low Autocorrelation Binary Sequences
• Evolving the Structure of Evolution Strategies
• COCO: Performance Assessment
• Automated Algorithm Selection: Survey and Perspectives
• The linear hidden subset problem for the (1+1) EA with scheduled and adaptive mutation rates
• Numerical Simulations of the Ising Model on the Union Jack Lattice

Cited by in corpus (12)

• Performance comparison of optimization methods on variational quantum algorithms
• Benchmarking in Optimization: Best Practice and Open Issues
• Automated Configuration of Genetic Algorithms by Tuning for Anytime Performance
• Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization
• Benchmarking a $(\mu +\lambda )$ Genetic Algorithm with Configurable Crossover Probability
• Towards Large Scale Automated Algorithm Design by Integrating Modular Benchmarking Frameworks
• Optimal Static Mutation Strength Distributions for the $(1+\lambda )$ Evolutionary Algorithm on OneMax
• OPTION: OPTImization Algorithm Benchmarking ONtology
• Fourier Analysis Meets Runtime Analysis: Precise Runtimes on Plateaus
• Blending Dynamic Programming with Monte Carlo Simulation for Bounding the Running Time of Evolutionary Algorithms
• Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection
• On the Assessment of Benchmark Suites for Algorithm Comparison