The Impact of Hyper-Parameter Tuning for Landscape-Aware Performance Regression and Algorithm Selection

Abstract

Automated algorithm selection and configuration methods that build on exploratory landscape analysis (ELA) are becoming very popular in Evolutionary Computation. However, despite a significantly growing number of applications, the underlying machine learning models are often chosen in an ad-hoc manner. We show in this work that three classical regression methods are able to achieve meaningful results for ELA-based algorithm selection. For those three models -- random forests, decision trees, and bagging decision trees -- the quality of the regression models is highly impacted by the chosen hyper-parameters. This has significant effects also on the quality of the algorithm selectors that are built on top of these regressions. By comparing a total number of 30 different models, each coupled with 2 complementary regression strategies, we derive guidelines for the tuning of the regression models and provide general recommendations for a more systematic use of classical machine learning models in landscape-aware algorithm selection. We point out that a choice of the machine learning model merits to be carefully undertaken and further investigated.

To appear in the Proceedings of Genetic and Evolutionary Computation Conference (GECCO 2021), ACM

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
• Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning
• IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics
• Algorithm Runtime Prediction: Methods & Evaluation
• Landscape-Aware Fixed-Budget Performance Regression and Algorithm Selection for Modular CMA-ES Variants
• Towards White-box Benchmarks for Algorithm Control

Cited by in corpus (5)

• Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization
• Automated Algorithm Selection for Radar Network Configuration
• A Survey of Features Used for Representing Black-box Single-objective Continuous Optimization
• Benchmarking the Hooke-Jeeves Method, MTS-LS1, and BSrr on the Large-scale BBOB Function Set
• On Constructing Algorithm Portfolios in Algorithm Selection for Computationally Expensive Black-box Optimization in the Fixed-budget Setting