Computationally Efficient High-Dimensional Bayesian Optimization via Variable Selection

Abstract

Bayesian Optimization (BO) is a method for globally optimizing black-box functions. While BO has been successfully applied to many scenarios, developing effective BO algorithms that scale to functions with high-dimensional domains is still a challenge. Optimizing such functions by vanilla BO is extremely time-consuming. Alternative strategies for high-dimensional BO that are based on the idea of embedding the high-dimensional space to the one with low dimension are sensitive to the choice of the embedding dimension, which needs to be pre-specified. We develop a new computationally efficient high-dimensional BO method that exploits variable selection. Our method is able to automatically learn axis-aligned sub-spaces, i.e. spaces containing selected variables, without the demand of any pre-specified hyperparameters. We theoretically analyze the computational complexity of our algorithm and derive the regret bound. We empirically show the efficacy of our method on several synthetic and real problems.

This work has already been accepted in AutoML 2023

References in corpus (9)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• A Tutorial on Bayesian Optimization
• Constrained Bayesian Optimization for Automatic Chemical Design
• High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces
• Bayesian Optimization for Synthetic Gene Design
• Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
• Variable selection for Gaussian processes via sensitivity analysis of the posterior predictive distribution
• Automated Machine Learning on Big Data using Stochastic Algorithm Tuning