BOHB: Robust and Efficient Hyperparameter Optimization at Scale

Abstract

Modern deep learning methods are very sensitive to many hyperparameters, and, due to the long training times of state-of-the-art models, vanilla Bayesian hyperparameter optimization is typically computationally infeasible. On the other hand, bandit-based configuration evaluation approaches based on random search lack guidance and do not converge to the best configurations as quickly. Here, we propose to combine the benefits of both Bayesian optimization and bandit-based methods, in order to achieve the best of both worlds: strong anytime performance and fast convergence to optimal configurations. We propose a new practical state-of-the-art hyperparameter optimization method, which consistently outperforms both Bayesian optimization and Hyperband on a wide range of problem types, including high-dimensional toy functions, support vector machines, feed-forward neural networks, Bayesian neural networks, deep reinforcement learning, and convolutional neural networks. Our method is robust and versatile, while at the same time being conceptually simple and easy to implement.

arXiv commentpublished at ICML2018

References in corpus (1)

• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

Cited by in corpus (14)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Efficient Automatic CASH via Rising Bandits
• Interval Load Forecasting for Individual Households in the Presence of Electric Vehicle Charging
• Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• Trainless Model Performance Estimation for Neural Architecture Search
• Automatic Machine Learning for Multi-Receiver CNN Technology Classifiers
• Super-convergence and Differential Privacy: Training faster with better privacy guarantees
• Early Diagnosis of Retinal Blood Vessel Damage via Deep Learning-Powered Collective Intelligence Models
• Black-box Mixed-Variable Optimisation using a Surrogate Model that Satisfies Integer Constraints
• Scalable Acceleration for Classification-Based Derivative-Free Optimization
• Deep Neural Networks for Choice Analysis: Architectural Design with Alternative-Specific Utility Functions