Autotune: A Derivative-free Optimization Framework for Hyperparameter Tuning

Abstract

Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problems, whose objective functions tend to be nonsmooth, discontinuous, unpredictably varying in computational expense, and include continuous, categorical, and/or integer variables. Further, function evaluations can fail for a variety of reasons including numerical difficulties or hardware failures. Additionally, not all hyperparameter value combinations are compatible, which creates so called hidden constraints. Robust and efficient optimization algorithms are needed for hyperparameter tuning. In this paper we present an automated parallel derivative-free optimization framework called \textbf{Autotune}, which combines a number of specialized sampling and search methods that are very effective in tuning machine learning models despite these challenges. Autotune provides significantly improved models over using default hyperparameter settings with minimal user interaction on real-world applications. Given the inherent expense of training numerous candidate models, we demonstrate the effectiveness of Autotune's search methods and the efficient distributed and parallel paradigms for training and tuning models, and also discuss the resource trade-offs associated with the ability to both distribute the training process and parallelize the tuning process.

10 Pages, 9 figures, accept by KDD 2018

References in corpus (2)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Stratified Analysis of Bayesian Optimization Methods

Cited by in corpus (12)

• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• Compact Model Parameter Extraction via Derivative-Free Optimization
• Hyperparameter Auto-tuning in Self-Supervised Robotic Learning
• Massively Parallel Genetic Optimization through Asynchronous Propagation of Populations
• Transfer Learning based Search Space Design for Hyperparameter Tuning
• SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees
• JITuNE: Just-In-Time Hyperparameter Tuning for Network Embedding Algorithms
• Robust Pareto Transistor Sizing of GaN HEMTs for Millimeter-Wave Applications
• ExperienceThinking: Constrained Hyperparameter Optimization based on Knowledge and Pruning
• Learning Sampling Policy for Faster Derivative Free Optimization
• ChaCha for Online AutoML
• An Extensive Experimental Evaluation of Automated Machine Learning Methods for Recommending Classification Algorithms (Extended Version)