Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization

Abstract

This paper presents a new type of hybrid model for Bayesian optimization (BO) adept at managing mixed variables, encompassing both quantitative (continuous and integer) and qualitative (categorical) types. Our proposed new hybrid models (named hybridM) merge the Monte Carlo Tree Search structure (MCTS) for categorical variables with Gaussian Processes (GP) for continuous ones. hybridM leverages the upper confidence bound tree search (UCTS) for MCTS strategy, showcasing the tree architecture's integration into Bayesian optimization. Our innovations, including dynamic online kernel selection in the surrogate modeling phase and a unique UCTS search strategy, position our hybrid models as an advancement in mixed-variable surrogate models. Numerical experiments underscore the superiority of hybrid models, highlighting their potential in Bayesian optimization.

arXiv comment33 pages, 8 Figures

References in corpus (17)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search: A Survey
• BART: Bayesian additive regression trees
• 3D-HST WFC3-selected Photometric Catalogs in the Five CANDELS/3D-HST Fields: Photometry, Photometric Redshifts and Stellar Masses
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Progressive Sampling-Based Bayesian Optimization for Efficient and Automatic Machine Learning Model Selection
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• Bayesian Optimisation over Multiple Continuous and Categorical Inputs
• Black-box Combinatorial Optimization using Models with Integer-valued Minima
• Bayesian Inference in Monte-Carlo Tree Search
• Multitask and Transfer Learning for Autotuning Exascale Applications
• Bayesian Optimization over Hybrid Spaces
• Non-smooth Bayesian Optimization in Tuning Problems
• Structural Kernel Search via Bayesian Optimization and Symbolical Optimal Transport
• Efficient and Robust Bayesian Selection of Hyperparameters in Dimension Reduction for Visualization
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels

Cited by in corpus (1)

• Compactly-supported nonstationary kernels for computing exact Gaussian processes on big data