Multi-fidelity Bayesian Optimisation with Continuous Approximations

Abstract

Bandit methods for black-box optimisation, such as Bayesian optimisation, are used in a variety of applications including hyper-parameter tuning and experiment design. Recently, \emph{multi-fidelity} methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. In many practical applications however, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data $N$ and/or few training iterations $T$. Here, the approximations are best viewed as arising out of a continuous two dimensional space $(N,T)$. In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.

References in corpus (9)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Scrutinizing Exotic Cosmological Models Using ESSENCE Supernova Data Combined with Other Cosmological Probes
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• High Dimensional Bayesian Optimisation and Bandits via Additive Models
• A Bayesian model selection analysis of WMAP3
• Exponential Regret Bounds for Gaussian Process Bandits with Deterministic Observations
• Bayesian Optimization for Synthetic Gene Design
• On the optimal estimation of probability measures in weak and strong topologies

Cited by in corpus (36)

• The frontier of simulation-based inference
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Random Search and Reproducibility for Neural Architecture Search
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Learning the Effect of Registration Hyperparameters with HyperMorph
• Cost-aware Bayesian Optimization
• Efficient Automatic CASH via Rising Bandits
• A Latent Variable Approach for Non-Hierarchical Multi-Fidelity Adaptive Sampling
• Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization
• Multi-Fidelity Bayesian Optimization via Deep Neural Networks
• TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning
• Output Space Entropy Search Framework for Multi-Objective Bayesian Optimization
• Leveraging Trust for Joint Multi-Objective and Multi-Fidelity Optimization
• Bayesian Optimization with Binary Auxiliary Information
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• GIBBON: General-purpose Information-Based Bayesian OptimisatioN
• Deep Multi-Fidelity Active Learning of High-dimensional Outputs
• Information-Theoretic Multi-Objective Bayesian Optimization with Continuous Approximations
• Meta-Surrogate Benchmarking for Hyperparameter Optimization
• Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs
• Multipoint-BAX: A New Approach for Efficiently Tuning Particle Accelerator Emittance via Virtual Objectives
• Pareto-efficient Acquisition Functions for Cost-Aware Bayesian Optimization
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• BOSH: Bayesian Optimization by Sampling Hierarchically
• Bayesian Optimization Algorithms for Accelerator Physics
• HyperSTAR: Task-Aware Hyperparameters for Deep Networks
• Zeroth Order Non-convex optimization with Dueling-Choice Bandits
• Adaptive Batching for Gaussian Process Surrogates with Application in Noisy Level Set Estimation
• Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
• BORE: Bayesian Optimization by Density-Ratio Estimation
• Noisy Blackbox Optimization with Multi-Fidelity Queries: A Tree Search Approach
• Fast Hyperparameter Optimization of Deep Neural Networks via Ensembling Multiple Surrogates
• Efficient Hyperparameter Optimization for Physics-based Character Animation
• Bayesian optimization for modular black-box systems with switching costs
• Nonmyopic Multifidelity Active Search