A Stratified Analysis of Bayesian Optimization Methods
arXiv:1603.09441
Abstract
Empirical analysis serves as an important complement to theoretical analysis for studying practical Bayesian optimization. Often empirical insights expose strengths and weaknesses inaccessible to theoretical analysis. We define two metrics for comparing the performance of Bayesian optimization methods and propose a ranking mechanism for summarizing performance within various genres or strata of test functions. These test functions serve to mimic the complexity of hyperparameter optimization problems, the most prominent application of Bayesian optimization, but with a closed form which allows for rapid evaluation and more predictable behavior. This offers a flexible and efficient way to investigate functions with specific properties of interest, such as oscillatory behavior or an optimum on the domain boundary.
References in corpus (6)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Scalable Bayesian Optimization Using Deep Neural Networks
- BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits
- Predictive Entropy Search for Bayesian Optimization with Unknown Constraints
- Input Warping for Bayesian Optimization of Non-stationary Functions
- GLASSES: Relieving The Myopia Of Bayesian Optimisation
Cited by in corpus (10)
- Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning
- Bayesian Optimization of Combinatorial Structures
- Using a thousand optimization tasks to learn hyperparameter search strategies
- Evaluation System for a Bayesian Optimization Service
- On the role of Model Uncertainties in Bayesian Optimization
- Bayesian Calibration and Uncertainty Quantification for a Large Nutrient Load Impact Model
- Correcting boundary over-exploration deficiencies in Bayesian optimization with virtual derivative sign observations
- High-dimensional Black-box Optimization Under Uncertainty
- Preferential Batch Bayesian Optimization
- Trust-Region Algorithms for Training Responses: Machine Learning Methods Using Indefinite Hessian Approximations