Model-based Asynchronous Hyperparameter and Neural Architecture Search
arXiv:2003.10865
Abstract
We introduce a model-based asynchronous multi-fidelity method for hyperparameter and neural architecture search that combines the strengths of asynchronous Hyperband and Gaussian process-based Bayesian optimization. At the heart of our method is a probabilistic model that can simultaneously reason across hyperparameters and resource levels, and supports decision-making in the presence of pending evaluations. We demonstrate the effectiveness of our method on a wide range of challenging benchmarks, for tabular data, image classification and language modelling, and report substantial speed-ups over current state-of-the-art methods. Our new methods, along with asynchronous baselines, are implemented in a distributed framework which will be open sourced along with this publication.
References in corpus (11)
- Practical Bayesian Optimization of Machine Learning Algorithms
- DARTS: Differentiable Architecture Search
- OpenML: networked science in machine learning
- Tune: A Research Platform for Distributed Model Selection and Training
- Large-Scale Evolution of Image Classifiers
- Regularizing and Optimizing LSTM Language Models
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Freeze-Thaw Bayesian Optimization
- Deep Gaussian Processes for Multi-fidelity Modeling
- Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation