Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
arXiv:1908.06674
Abstract
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of optimizing BO's own hyperparameters and the transferability of the found settings using a wide range of benchmarks, including artificial functions, HPO and HPO combined with neural architecture search. In particular, we show (i) that tuning can improve the any-time performance of different BO approaches, that optimized BO settings also perform well (ii) on similar problems and (iii) partially even on problems from other problem families, and (iv) which BO hyperparameters are most important.
Accepted at DSO workshop (as part of IJCAI'19)
References in corpus (10)
- Practical Bayesian Optimization of Machine Learning Algorithms
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Scalable Bayesian Optimization Using Deep Neural Networks
- Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
- A Tutorial on Bayesian Optimization
- Structure Discovery in Nonparametric Regression through Compositional Kernel Search
- Learning to Learn without Gradient Descent by Gradient Descent
- The Parallel Knowledge Gradient Method for Batch Bayesian Optimization
- Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization
- Automatic Exploration of Machine Learning Experiments on OpenML