Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters

Abstract

Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to find the best optima. Surprisingly, there is little theoretical analysis of this crucial problem in the literature. To address this, we derive a cumulative regret bound for Bayesian optimisation with Gaussian processes and unknown kernel hyper-parameters in the stochastic setting. The bound, which applies to the expected improvement acquisition function and sub-Gaussian observation noise, provides us with guidelines on how to design hyper-parameter estimation methods. A simple simulation demonstrates the importance of following these guidelines.

16 pages, 1 figure

References in corpus (1)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

Cited by in corpus (14)

• Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation
• Heteroscedastic Treed Bayesian Optimisation
• Streaming kernel regression with provably adaptive mean, variance, and regularization
• Knowing The What But Not The Where in Bayesian Optimization
• PARyOpt: A software for Parallel Asynchronous Remote Bayesian Optimization
• A Domain-Shrinking based Bayesian Optimization Algorithm with Order-Optimal Regret Performance
• Optimal Order Simple Regret for Gaussian Process Bandits
• Practical Batch Bayesian Optimization for Less Expensive Functions
• Bayesian Optimization for Iterative Learning
• No-Regret Bayesian Optimization with Unknown Hyperparameters
• Evolving Gaussian Process kernels from elementary mathematical expressions
• Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systems
• An iterative scheme for feature based positioning using a weighted dissimilarity measure
• Recursive Two-Step Lookahead Expected Payoff for Time-Dependent Bayesian Optimization