HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
arXiv:2109.06716
Abstract
To achieve peak predictive performance, hyperparameter optimization (HPO) is a crucial component of machine learning and its applications. Over the last years, the number of efficient algorithms and tools for HPO grew substantially. At the same time, the community is still lacking realistic, diverse, computationally cheap, and standardized benchmarks. This is especially the case for multi-fidelity HPO methods. To close this gap, we propose HPOBench, which includes 7 existing and 5 new benchmark families, with a total of more than 100 multi-fidelity benchmark problems. HPOBench allows to run this extendable set of multi-fidelity HPO benchmarks in a reproducible way by isolating and packaging the individual benchmarks in containers. It also provides surrogate and tabular benchmarks for computationally affordable yet statistically sound evaluations. To demonstrate HPOBench's broad compatibility with various optimization tools, as well as its usefulness, we conduct an exemplary large-scale study evaluating 13 optimizers from 6 optimization tools. We provide HPOBench here: https://github.com/automl/HPOBench.
arXiv commentPublished at NeurIPS Datasets and Benchmarks Track 2021. Updated version
References in corpus (20)
- Scikit-learn: Machine Learning in Python
- Practical Bayesian Optimization of Machine Learning Algorithms
- OpenML: networked science in machine learning
- Classifier Technology and the Illusion of Progress
- Stochastic Gradient Hamiltonian Monte Carlo
- BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
- Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
- Freeze-Thaw Bayesian Optimization
- Multi-fidelity Bayesian Optimisation with Continuous Approximations
- SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
- Benchmarking in Optimization: Best Practice and Open Issues
- A Critical Assessment of Benchmark Comparison in Planning
- A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes
- IOHprofiler: A Benchmarking and Profiling Tool for Iterative Optimization Heuristics
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
- Differential Evolution for Neural Architecture Search
- Using a thousand optimization tasks to learn hyperparameter search strategies
- A Kernel for Hierarchical Parameter Spaces
- Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameters
- Data vs classifiers, who wins?