Parallel Architecture and Hyperparameter Search via Successive Halving and Classification

Abstract

We present a simple and powerful algorithm for parallel black box optimization called Successive Halving and Classification (SHAC). The algorithm operates in $K$ stages of parallel function evaluations and trains a cascade of binary classifiers to iteratively cull the undesirable regions of the search space. SHAC is easy to implement, requires no tuning of its own configuration parameters, is invariant to the scale of the objective function and can be built using any choice of binary classifier. We adopt tree-based classifiers within SHAC and achieve competitive performance against several strong baselines for optimizing synthetic functions, hyperparameters and architectures.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Progressive Neural Architecture Search
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Accelerating Neural Architecture Search using Performance Prediction

Cited by in corpus (9)

• AutoAugment: Learning Augmentation Policies from Data
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Feature Partitioning for Efficient Multi-Task Architectures
• Reusing Trained Layers of Convolutional Neural Networks to Shorten Hyperparameters Tuning Time
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Reducing The Search Space For Hyperparameter Optimization Using Group Sparsity
• MOFA: Modular Factorial Design for Hyperparameter Optimization
• Pathological Voice Classification Using Mel-Cepstrum Vectors and Support Vector Machine