Lifelong Bayesian Optimization
arXiv:1905.12280
Abstract
Automatic Machine Learning (Auto-ML) systems tackle the problem of automating the design of prediction models or pipelines for data science. In this paper, we present Lifelong Bayesian Optimization (LBO), an online, multitask Bayesian optimization (BO) algorithm designed to solve the problem of model selection for datasets arriving and evolving over time. To be suitable for "lifelong" Bayesian Optimization, an algorithm needs to scale with the ever increasing number of acquisitions and should be able to leverage past optimizations in learning the current best model. We cast the problem of model selection as a black-box function optimization problem. In LBO, we exploit the correlation between functions by using components of previously learned functions to speed up the learning process for newly arriving datasets. Experiments on real and synthetic data show that LBO outperforms standard BO algorithms applied repeatedly on the data.
17 pages, 8 figures
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Scikit-learn: Machine Learning in Python
- Overcoming catastrophic forgetting in neural networks
- Practical Bayesian Optimization of Machine Learning Algorithms
- Categorical Reparameterization with Gumbel-Softmax
- Continual Learning Through Synaptic Intelligence
- Stochastic Gradient Hamiltonian Monte Carlo
- Scalable Bayesian Optimization Using Deep Neural Networks
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
- The Benefit of Multitask Representation Learning
- Thoughts on Massively Scalable Gaussian Processes
- A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes
- AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning
- Stick-Breaking Variational Autoencoders