High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces

Abstract

Bayesian optimization (BO) is a powerful paradigm for efficient optimization of black-box objective functions. High-dimensional BO presents a particular challenge, in part because the curse of dimensionality makes it difficult to define -- as well as do inference over -- a suitable class of surrogate models. We argue that Gaussian process surrogate models defined on sparse axis-aligned subspaces offer an attractive compromise between flexibility and parsimony. We demonstrate that our approach, which relies on Hamiltonian Monte Carlo for inference, can rapidly identify sparse subspaces relevant to modeling the unknown objective function, enabling sample-efficient high-dimensional BO. In an extensive suite of experiments comparing to existing methods for high-dimensional BO we demonstrate that our algorithm, Sparse Axis-Aligned Subspace BO (SAASBO), achieves excellent performance on several synthetic and real-world problems without the need to set problem-specific hyperparameters.

arXiv commentTo appear in UAI 2021

References in corpus (7)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization
• Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro
• Scalable Global Optimization via Local Bayesian Optimization
• BOCK : Bayesian Optimization with Cylindrical Kernels
• Bayesian Optimization of Combinatorial Structures
• Combinatorial Bayesian Optimization using the Graph Cartesian Product

Cited by in corpus (10)

• Investigating Bayesian optimization for expensive-to-evaluate black box functions: Application in fluid dynamics
• High-dimensional Bayesian Optimization of Hyperparameters for an Attention-based Network to Predict Materials Property: a Case Study on CrabNet using Ax and SAASBO
• Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces
• Bayesian Optimization Algorithms for Accelerator Physics
• Computationally Efficient High-Dimensional Bayesian Optimization via Variable Selection
• Bayesian Optimization for Ion Beam Centroid Correction
• Performance optimization for a scintillating glass electromagnetic calorimeter at the EIC
• Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB
• RTDK-BO: High Dimensional Bayesian Optimization with Reinforced Transformer Deep kernels
• High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling