Non-smooth Bayesian Optimization in Tuning Problems
arXiv:2109.07563
Abstract
Building surrogate models is one common approach when we attempt to learn unknown black-box functions. Bayesian optimization provides a framework which allows us to build surrogate models based on sequential samples drawn from the function and find the optimum. Tuning algorithmic parameters to optimize the performance of large, complicated "black-box" application codes is a specific important application, which aims at finding the optima of black-box functions. Within the Bayesian optimization framework, the Gaussian process model produces smooth or continuous sample paths. However, the black-box function in the tuning problem is often non-smooth. This difficult tuning problem is worsened by the fact that we usually have limited sequential samples from the black-box function. Motivated by these issues encountered in tuning, we propose a novel additive Gaussian process model called clustered Gaussian process (cGP), where the additive components are induced by clustering. In the examples we studied, the performance can be improved by as much as 90% among repetitive experiments. By using this surrogate model, we want to capture the non-smoothness of the black-box function. In addition to an algorithm for constructing this model, we also apply the model to several artificial and real applications to evaluate it.
arXiv comment61 pages
References in corpus (13)
- Practical Bayesian Optimization of Machine Learning Algorithms
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Bayesian Online Changepoint Detection
- Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
- Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
- Student-t Processes as Alternatives to Gaussian Processes
- Freeze-Thaw Bayesian Optimization
- Input Warping for Bayesian Optimization of Non-stationary Functions
- A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes
- Cost-aware Bayesian Optimization
- Sparse Additive Gaussian Process Regression
- Multitask and Transfer Learning for Autotuning Exascale Applications
- Particle learning of Gaussian process models for sequential design and optimization