Benchmarking in Optimization: Best Practice and Open Issues

Abstract

This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.

Version 2

References in corpus (12)

• ParamILS: An Automatic Algorithm Configuration Framework
• SATzilla: Portfolio-based Algorithm Selection for SAT
• An Easy-to-use Real-world Multi-objective Optimization Problem Suite
• A new Taxonomy of Continuous Global Optimization Algorithms
• Benchmarking Discrete Optimization Heuristics with IOHprofiler
• IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics
• Better Software Analytics via "DUO": Data Mining Algorithms Using/Used-by Optimizers
• Critical Hyper-Parameters: No Random, No Cry
• On the Effectiveness of Simple Success-Based Parameter Selection Mechanisms for Two Classical Discrete Black-Box Optimization Benchmark Problems
• Black-box Mixed-Variable Optimisation using a Surrogate Model that Satisfies Integer Constraints
• Noisy Optimization: Fast Convergence Rates with Comparison-Based Algorithms
• Versatile Black-Box Optimization

Cited by in corpus (16)

• Challenges and Opportunities in Quantum Optimization
• A Survey on Learnable Evolutionary Algorithms for Scalable Multiobjective Optimization
• Metaheuristics "In the Large"
• Characterization of Constrained Continuous Multiobjective Optimization Problems: A Feature Space Perspective
• CAAI -- A Cognitive Architecture to Introduce Artificial Intelligence in Cyber-Physical Production Systems
• Experimental evaluation of architectural software performance design patterns in microservices
• EXPObench: Benchmarking Surrogate-based Optimisation Algorithms on Expensive Black-box Functions
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
• Tools for Landscape Analysis of Optimisation Problems in Procedural Content Generation for Games
• Computational Rational Engineering and Development: Synergies and Opportunities
• Generating Instances with Performance Differences for More Than Just Two Algorithms
• A Continuous Optimisation Benchmark Suite from Neural Network Regression
• Backtracking (the) Algorithms on the Hamiltonian Cycle Problem
• Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT
• On the Assessment of Benchmark Suites for Algorithm Comparison