Combinatorial Bayesian Optimization using the Graph Cartesian Product

Abstract

This paper focuses on Bayesian Optimization (BO) for objectives on combinatorial search spaces, including ordinal and categorical variables. Despite the abundance of potential applications of Combinatorial BO, including chipset configuration search and neural architecture search, only a handful of methods have been proposed. We introduce COMBO, a new Gaussian Process (GP) BO. COMBO quantifies "smoothness" of functions on combinatorial search spaces by utilizing a combinatorial graph. The vertex set of the combinatorial graph consists of all possible joint assignments of the variables, while edges are constructed using the graph Cartesian product of the sub-graphs that represent the individual variables. On this combinatorial graph, we propose an ARD diffusion kernel with which the GP is able to model high-order interactions between variables leading to better performance. Moreover, using the Horseshoe prior for the scale parameter in the ARD diffusion kernel results in an effective variable selection procedure, making COMBO suitable for high dimensional problems. Computationally, in COMBO the graph Cartesian product allows the Graph Fourier Transform calculation to scale linearly instead of exponentially. We validate COMBO in a wide array of realistic benchmarks, including weighted maximum satisfiability problems and neural architecture search. COMBO outperforms consistently the latest state-of-the-art while maintaining computational and statistical efficiency.

arXiv commentAccepted to NeurIPS 2019, code: https://github.com/QUVA-Lab/COMBO

Cited by in corpus (11)

• High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces
• Scalable Constrained Bayesian Optimization
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces
• Output Space Entropy Search Framework for Multi-Objective Bayesian Optimization
• Scalable Combinatorial Bayesian Optimization with Tractable Statistical models
• Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces
• Bayesian Optimization over Hybrid Spaces
• Constrained Discrete Black-Box Optimization using Mixed-Integer Programming
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
• Explaining Inference Queries with Bayesian Optimization