Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB

Abstract

Bayesian Optimization (BO) is a class of black-box, surrogate-based heuristics that can efficiently optimize problems that are expensive to evaluate, and hence admit only small evaluation budgets. BO is particularly popular for solving numerical optimization problems in industry, where the evaluation of objective functions often relies on time-consuming simulations or physical experiments. However, many industrial problems depend on a large number of parameters. This poses a challenge for BO algorithms, whose performance is often reported to suffer when the dimension grows beyond 15 variables. Although many new algorithms have been proposed to address this problem, it is not well understood which one is the best for which optimization scenario. In this work, we compare five state-of-the-art high-dimensional BO algorithms, with vanilla BO and CMA-ES on the 24 BBOB functions of the COCO environment at increasing dimensionality, ranging from 10 to 60 variables. Our results confirm the superiority of BO over CMA-ES for limited evaluation budgets and suggest that the most promising approach to improve BO is the use of trust regions. However, we also observe significant performance differences for different function landscapes and budget exploitation phases, indicating improvement potential, e.g., through hybridization of algorithmic components.

References in corpus (13)

• COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
• Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• Auto-Keras: An Efficient Neural Architecture Search System
• IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics
• High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces
• Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
• HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation
• Additive Kernels for Gaussian Process Modeling
• Randomly Projected Additive Gaussian Processes for Regression
• Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces
• Semi-supervised Embedding Learning for High-dimensional Bayesian Optimization
• Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?