Cost-aware Bayesian Optimization

Abstract

Bayesian optimization (BO) is a class of global optimization algorithms, suitable for minimizing an expensive objective function in as few function evaluations as possible. While BO budgets are typically given in iterations, this implicitly measures convergence in terms of iteration count and assumes each evaluation has identical cost. In practice, evaluation costs may vary in different regions of the search space. For example, the cost of neural network training increases quadratically with layer size, which is a typical hyperparameter. Cost-aware BO measures convergence with alternative cost metrics such as time, energy, or money, for which vanilla BO methods are unsuited. We introduce Cost Apportioned BO (CArBO), which attempts to minimize an objective function in as little cost as possible. CArBO combines a cost-effective initial design with a cost-cooled optimization phase which depreciates a learned cost model as iterations proceed. On a set of 20 black-box function optimization problems we show that, given the same cost budget, CArBO finds significantly better hyperparameter configurations than competing methods.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Practical Bayesian Optimization of Machine Learning Algorithms
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• A Tutorial on Bayesian Optimization
• Max-value Entropy Search for Efficient Bayesian Optimization
• Numerical studies of space filling designs: optimization of Latin Hypercube Samples and subprojection properties
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Multi-fidelity Bayesian Optimisation with Continuous Approximations
• OBOE: Collaborative Filtering for AutoML Model Selection
• Maximizing acquisition functions for Bayesian optimization
• Cost-aware Multi-objective Bayesian optimisation

Cited by in corpus (7)

• Amazon SageMaker Autopilot: a white box AutoML solution at scale
• Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs
• Pareto-efficient Acquisition Functions for Cost-Aware Bayesian Optimization
• Non-smooth Bayesian Optimization in Tuning Problems
• A Nonmyopic Approach to Cost-Constrained Bayesian Optimization
• MISO-wiLDCosts: Multi Information Source Optimization with Location Dependent Costs
• Bayesian optimization for modular black-box systems with switching costs