Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020

Abstract

This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.

References in corpus (18)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization
• Bayesian optimization for materials design
• Batch Bayesian Optimization via Local Penalization
• BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
• Scalable Global Optimization via Local Bayesian Optimization
• The Parallel Knowledge Gradient Method for Batch Bayesian Optimization
• Predictive Entropy Search for Bayesian Optimization with Unknown Constraints
• pySOT and POAP: An event-driven asynchronous framework for surrogate optimization
• GPflowOpt: A Bayesian Optimization Library using TensorFlow
• Emulation of physical processes with Emukit
• Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search
• Better call Surrogates: A hybrid Evolutionary Algorithm for Hyperparameter optimization
• ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language
• Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization
• Squirrel: A Switching Hyperparameter Optimizer
• GPU Accelerated Exhaustive Search for Optimal Ensemble of Black-Box Optimization Algorithms
• Adaptive Local Bayesian Optimization Over Multiple Discrete Variables

Cited by in corpus (18)

• Revisiting Deep Learning Models for Tabular Data
• Language Model Crossover: Variation through Few-Shot Prompting
• Learning-based Design and Control for Quadrupedal Robots with Parallel-Elastic Actuators
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• On the Feasibility of Cross-Language Detection of Malicious Packages in npm and PyPI
• Optimizing Facial Expressions of an Android Robot Effectively: a Bayesian Optimization Approach
• Bayesian Optimisation for Sequential Experimental Design with Applications in Additive Manufacturing
• Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces
• Optimizing with Low Budgets: a Comparison on the Black-box Optimization Benchmarking Suite and OpenAI Gym
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Performance versus Complexity Study of Neural Network Equalizers in Coherent Optical Systems
• Latency-Aware Neural Architecture Search with Multi-Objective Bayesian Optimization
• Experimenting, Fast and Slow: Bayesian Optimization of Long-term Outcomes with Online Experiments
• Predicting Foreign Exchange EUR/USD direction using machine learning
• On the role of Model Uncertainties in Bayesian Optimization
• Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOB
• A Nonmyopic Approach to Cost-Constrained Bayesian Optimization
• Automated machine learning for borehole resistivity measurements