High Dimensional Bayesian Optimisation and Bandits via Additive Models

Abstract

Bayesian Optimisation (BO) is a technique used in optimising a $D$-dimensional function which is typically expensive to evaluate. While there have been many successes for BO in low dimensions, scaling it to high dimensions has been notoriously difficult. Existing literature on the topic are under very restrictive settings. In this paper, we identify two key challenges in this endeavour. We tackle these challenges by assuming an additive structure for the function. This setting is substantially more expressive and contains a richer class of functions than previous work. We prove that, for additive functions the regret has only linear dependence on $D$ even though the function depends on all $D$ dimensions. We also demonstrate several other statistical and computational benefits in our framework. Via synthetic examples, a scientific simulation and a face detection problem we demonstrate that our method outperforms naive BO on additive functions and on several examples where the function is not additive.

Proceedings of The 32nd International Conference on Machine Learning 2015

References in corpus (6)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Cosmological Constraints from the SDSS Luminous Red Galaxies
• A Bayesian model selection analysis of WMAP3
• Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations
• Bayesian Optimization for Synthetic Gene Design

Cited by in corpus (25)

• A Tutorial on Bayesian Optimization
• Autonomous discovery of battery electrolytes with robotic experimentation and machine-learning
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• Multi-fidelity Bayesian Optimisation with Continuous Approximations
• Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models
• Bayesian Optimization with Exponential Convergence
• Bayesian optimisation for likelihood-free cosmological inference
• Upper Trust Bound Feasibility Criterion for Mixed Constrained Bayesian Optimization with Application to Aircraft Design
• ENTMOOT: A Framework for Optimization over Ensemble Tree Models
• Event generator tuning using Bayesian optimization
• Bayesian Optimization with Adaptive Kernels for Robot Control
• Time-Varying Gaussian Process Bandit Optimization
• Query Efficient Posterior Estimation in Scientific Experiments via Bayesian Active Learning
• Asynchronous Parallel Bayesian Optimisation via Thompson Sampling
• Deep Kernel and Image Quality Estimators for Optimizing Robotic Ultrasound Controller using Bayesian Optimization
• Bayesian Optimisation over Multiple Continuous and Categorical Inputs
• Additive Tree-Structured Conditional Parameter Spaces in Bayesian Optimization: A Novel Covariance Function and a Fast Implementation
• Bayesian Optimization with Dimension Scheduling: Application to Biological Systems
• Efficient Online Hyperparameter Optimization for Kernel Ridge Regression with Applications to Traffic Time Series Prediction
• Rapid Bayesian optimisation for synthesis of short polymer fiber materials
• Regret bounds for meta Bayesian optimization with an unknown Gaussian process prior
• MBORE: Multi-objective Bayesian Optimisation by Density-Ratio Estimation
• Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systems
• Human-in-the-Loop Robot Planning with Non-Contextual Bandit Feedback
• Scalable Bayesian Optimization with Sparse Gaussian Process Models