Open Loop Hyperparameter Optimization and Determinantal Point Processes

Abstract

Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampling distributions. In particular, we propose the use of $k$-determinantal point processes in hyperparameter optimization via random search. Compared to conventional uniform random search where hyperparameter settings are sampled independently, a $k$-DPP promotes diversity. We describe an approach that transforms hyperparameter search spaces for efficient use with a $k$-DPP. In addition, we introduce a novel Metropolis-Hastings algorithm which can sample from $k$-DPPs defined over any space from which uniform samples can be drawn, including spaces with a mixture of discrete and continuous dimensions or tree structure. Our experiments show significant benefits in realistic scenarios with a limited budget for training supervised learners, whether in serial or parallel.

References in corpus (8)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Batch Bayesian Optimization via Local Penalization
• A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification
• Freeze-Thaw Bayesian Optimization
• Batched Gaussian Process Bandit Optimization via Determinantal Point Processes
• Critical Hyper-Parameters: No Random, No Cry
• Exact Sampling from Determinantal Point Processes

Cited by in corpus (1)

• GIBBON: General-purpose Information-Based Bayesian OptimisatioN