Weighting Is Worth the Wait: Bayesian Optimization with Importance Sampling

Abstract

Many contemporary machine learning models require extensive tuning of hyperparameters to perform well. A variety of methods, such as Bayesian optimization, have been developed to automate and expedite this process. However, tuning remains extremely costly as it typically requires repeatedly fully training models. We propose to accelerate the Bayesian optimization approach to hyperparameter tuning for neural networks by taking into account the relative amount of information contributed by each training example. To do so, we leverage importance sampling (IS); this significantly increases the quality of the black-box function evaluations, but also their runtime, and so must be done carefully. Casting hyperparameter search as a multi-task Bayesian optimization problem over both hyperparameters and importance sampling design achieves the best of both worlds: by learning a parameterization of IS that trades-off evaluation complexity and quality, we improve upon Bayesian optimization state-of-the-art runtime and final validation error across a variety of datasets and complex neural architectures.

References in corpus (6)

• Wide Residual Networks
• Predictive Entropy Search for Multi-objective Bayesian Optimization
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Freeze-Thaw Bayesian Optimization
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning

Cited by in corpus (1)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges