Towards Automatic Bayesian Optimization: A first step involving acquisition functions

Abstract

Bayesian Optimization is the state of the art technique for the optimization of black boxes, i.e., functions where we do not have access to their analytical expression nor its gradients, they are expensive to evaluate and its evaluation is noisy. The most popular application of bayesian optimization is the automatic hyperparameter tuning of machine learning algorithms, where we obtain the best configuration of machine learning algorithms by optimizing the estimation of the generalization error of these algorithms. Despite being applied with success, bayesian optimization methodologies also have hyperparameters that need to be configured such as the probabilistic surrogate model or the acquisition function used. A bad decision over the configuration of these hyperparameters implies obtaining bad quality results. Typically, these hyperparameters are tuned by making assumptions of the objective function that we want to evaluate but there are scenarios where we do not have any prior information about the objective function. In this paper, we propose a first attempt over automatic bayesian optimization by exploring several heuristics that automatically tune the acquisition function of bayesian optimization. We illustrate the effectiveness of these heurisitcs in a set of benchmark problems and a hyperparameter tuning problem of a machine learning algorithm.

References in corpus (7)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Max-value Entropy Search for Efficient Bayesian Optimization
• Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
• Suggesting Cooking Recipes Through Simulation and Bayesian Optimization
• An Artificial Consciousness Model and its relations with Philosophy of Mind