Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?

Abstract

Bayesian optimisation presents a sample-efficient methodology for global optimisation. Within this framework, a crucial performance-determining subroutine is the maximisation of the acquisition function, a task complicated by the fact that acquisition functions tend to be non-convex and thus nontrivial to optimise. In this paper, we undertake a comprehensive empirical study of approaches to maximise the acquisition function. Additionally, by deriving novel, yet mathematically equivalent, compositional forms for popular acquisition functions, we recast the maximisation task as a compositional optimisation problem, allowing us to benefit from the extensive literature in this field. We highlight the empirical advantages of the compositional approach to acquisition function maximisation across 3958 individual experiments comprising synthetic optimisation tasks as well as tasks from Bayesmark. Given the generality of the acquisition function maximisation subroutine, we posit that the adoption of compositional optimisers has the potential to yield performance improvements across all domains in which Bayesian optimisation is currently being applied.

References in corpus (33)

• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Practical Bayesian Optimization of Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• pymoo: Multi-objective Optimization in Python
• Expectation Propagation for approximate Bayesian inference
• A Literature Survey of Benchmark Functions For Global Optimization Problems
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Gaussian Processes for Big Data
• A Tutorial on Bayesian Optimization
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
• Max-value Entropy Search for Efficient Bayesian Optimization
• RES: Regularized Stochastic BFGS Algorithm
• Global Convergence of Online Limited Memory BFGS
• Stochastic simultaneous optimistic optimization
• Maximizing acquisition functions for Bayesian optimization
• A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes
• GPflowOpt: A Bayesian Optimization Library using TensorFlow
• Black-box optimization of noisy functions with unknown smoothness
• Gaussian Probabilities and Expectation Propagation
• Optimization of circuitry arrangements for heat exchangers using derivative-free optimization
• Wasserstein Robust Reinforcement Learning
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Gaussian Process Molecule Property Prediction with FlowMO
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Bayesian Optimisation over Multiple Continuous and Categorical Inputs
• Compositional ADAM: An Adaptive Compositional Solver
• Regret minimization in stochastic non-convex learning via a proximal-gradient approach
• Distributed SDDM Solvers: Theory & Applications

Cited by in corpus (4)

• Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge
• Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning
• High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
• Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation