A System for Massively Parallel Hyperparameter Tuning
arXiv:1810.05934
Abstract
Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed computing settings. We address this challenge by first introducing a simple and robust hyperparameter optimization algorithm called ASHA, which exploits parallelism and aggressive early-stopping to tackle large-scale hyperparameter optimization problems. Our extensive empirical results show that ASHA outperforms existing state-of-the-art hyperparameter optimization methods; scales linearly with the number of workers in distributed settings; and is suitable for massive parallelism, as demonstrated on a task with 500 workers. We then describe several design decisions we encountered, along with our associated solutions, when integrating ASHA in Determined AI's end-to-end production-quality machine learning system that offers hyperparameter tuning as a service.
arXiv commentv2: Corrected typo in Algorithm 1 v3: Added comparison to BOHB and parallel version of synchronous SHA. Add PBT to experiment in Section 4.3.1 v4: Added acknowledgements and slight edit to related work
References in corpus (9)
- Practical Bayesian Optimization of Machine Learning Algorithms
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Recurrent Neural Network Regularization
- DARTS: Differentiable Architecture Search
- One weird trick for parallelizing convolutional neural networks
- Tune: A Research Platform for Distributed Model Selection and Training
- Random Search and Reproducibility for Neural Architecture Search
- Batch Bayesian Optimization via Local Penalization
- A Generalized Framework for Population Based Training
Cited by in corpus (5)
- Deep learning-based surrogate model for 3-D patient-specific computational fluid dynamics
- Interpretable multiscale Machine Learning-Based Parameterizations of Convection for ICON
- Solving the scalarization issues of Advantage-based Reinforcement Learning Algorithms
- Fisher Task Distance and Its Application in Neural Architecture Search
- Optimizing Neural Network Scale for ECG Classification