Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

Abstract

Due to the high computational demands executing a rigorous comparison between hyperparameter optimization (HPO) methods is often cumbersome. The goal of this paper is to facilitate a better empirical evaluation of HPO methods by providing benchmarks that are cheap to evaluate, but still represent realistic use cases. We believe these benchmarks provide an easy and efficient way to conduct reproducible experiments for neural hyperparameter search. Our benchmarks consist of a large grid of configurations of a feed forward neural network on four different regression datasets including architectural hyperparameters and hyperparameters concerning the training pipeline. Based on this data, we performed an in-depth analysis to gain a better understanding of the properties of the optimization problem, as well as of the importance of different types of hyperparameters. Second, we exhaustively compared various different state-of-the-art methods from the hyperparameter optimization literature on these benchmarks in terms of performance and robustness.

References in corpus (6)

• Sequence to Sequence Learning with Neural Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• DARTS: Differentiable Architecture Search
• Large-Scale Evolution of Image Classifiers
• Designing Neural Network Architectures using Reinforcement Learning
• NAS-Bench-101: Towards Reproducible Neural Architecture Search

Cited by in corpus (24)

• AutoML: A Survey of the State-of-the-Art
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Best Practices for Scientific Research on Neural Architecture Search
• SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• AutoHAS: Efficient Hyperparameter and Architecture Search
• Differential Evolution for Neural Architecture Search
• Using a thousand optimization tasks to learn hyperparameter search strategies
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Meta-Surrogate Benchmarking for Hyperparameter Optimization
• Learning to Rank Learning Curves
• DAAS: Differentiable Architecture and Augmentation Policy Search
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Hyperparameter Transfer Learning with Adaptive Complexity
• Hyperparameter Transfer Across Developer Adjustments
• BORE: Bayesian Optimization by Density-Ratio Estimation
• A multi-objective perspective on jointly tuning hardware and hyperparameters
• Efficient Model Performance Estimation via Feature Histories
• Efficient Rollout Strategies for Bayesian Optimization
• JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data
• A resource-efficient method for repeated HPO and NAS problems
• Approximate Bayesian Optimisation for Neural Networks
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor