A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

Abstract

We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences.

References in corpus (2)

• PEGASUS: A Policy Search Method for Large MDPs and POMDPs
• Portfolio Allocation for Bayesian Optimization

Cited by in corpus (26)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Robots that can adapt like animals
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits
• Freeze-Thaw Bayesian Optimization
• Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms
• Portfolio Allocation for Bayesian Optimization
• Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization
• Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations
• An Entropy Search Portfolio for Bayesian Optimization
• Bayesian Optimization for Synthetic Gene Design
• Theoretical Analysis of Bayesian Optimisation with Unknown Gaussian Process Hyper-Parameters
• Joint Optimization and Variable Selection of High-dimensional Gaussian Processes
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
• Adaptive Hamiltonian and Riemann Manifold Monte Carlo Samplers
• Heteroscedastic Treed Bayesian Optimisation
• Automated Machine Learning on Big Data using Stochastic Algorithm Tuning
• Active Learning for Matching Problems
• Regret Bounds for Deterministic Gaussian Process Bandits
• Bayesian Optimisation for Machine Translation
• A Nonparametric Conjugate Prior Distribution for the Maximizing Argument of a Noisy Function
• Decentralized, Adaptive, Look-Ahead Particle Filtering
• apsis - Framework for Automated Optimization of Machine Learning Hyper Parameters
• Fast and optimal nonparametric sequential design for astronomical observations
• Classification and Bayesian Optimization for Likelihood-Free Inference
• Gaussian Process Bandits for Tree Search: Theory and Application to Planning in Discounted MDPs