Practical Transfer Learning for Bayesian Optimization

Abstract

When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extensions to Bayesian optimization and establish a worst-case bound compared to vanilla Bayesian optimization. Using a large collection of hyperparameter optimization benchmark problems, we demonstrate that our contributions substantially reduce optimization time compared to standard Gaussian process-based Bayesian optimization and improve over the current state-of-the-art for transfer hyperparameter optimization.

This version fixes a minor error in the equation in Section 3.2 of V3

References in corpus (11)

• Practical Bayesian Optimization of Machine Learning Algorithms
• OpenML: networked science in machine learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• A Tutorial on Bayesian Optimization
• Max-value Entropy Search for Efficient Bayesian Optimization
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• Maximizing acquisition functions for Bayesian optimization
• Probabilistic Matrix Factorization for Automated Machine Learning
• Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning
• Hyperparameter Learning via Distributional Transfer
• Automatic Exploration of Machine Learning Experiments on OpenML