A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes

Abstract

How can we efficiently gather information to optimize an unknown function, when presented with multiple, mutually dependent information sources with different costs? For example, when optimizing a robotic system, intelligently trading off computer simulations and real robot testings can lead to significant savings. Existing methods, such as multi-fidelity GP-UCB or Entropy Search-based approaches, either make simplistic assumptions on the interaction among different fidelities or use simple heuristics that lack theoretical guarantees. In this paper, we study multi-fidelity Bayesian optimization with complex structural dependencies among multiple outputs, and propose MF-MI-Greedy, a principled algorithmic framework for addressing this problem. In particular, we model different fidelities using additive Gaussian processes based on shared latent structures with the target function. Then we use cost-sensitive mutual information gain for efficient Bayesian global optimization. We propose a simple notion of regret which incorporates the cost of different fidelities, and prove that MF-MI-Greedy achieves low regret. We demonstrate the strong empirical performance of our algorithm on both synthetic and real-world datasets.

Cited by in corpus (26)

• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• A Latent Variable Approach for Non-Hierarchical Multi-Fidelity Adaptive Sampling
• Multi-fidelity Bayesian Optimization with Max-value Entropy Search and its parallelization
• Multi-Fidelity Bayesian Optimization via Deep Neural Networks
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• Scalable Combinatorial Bayesian Optimization with Tractable Statistical models
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Corruption-Tolerant Gaussian Process Bandit Optimization
• Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation
• Deep Multi-Fidelity Active Learning of High-dimensional Outputs
• A New Knowledge Gradient-based Method for Constrained Bayesian Optimization
• Lifelong Bayesian Optimization
• Information-Theoretic Multi-Objective Bayesian Optimization with Continuous Approximations
• Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs
• Optimizing Photonic Nanostructures via Multi-fidelity Gaussian Processes
• Multi-Fidelity High-Order Gaussian Processes for Physical Simulation
• Non-smooth Bayesian Optimization in Tuning Problems
• Bayesian Optimization for Iterative Learning
• Bayesian Screening: Multi-test Bayesian Optimization Applied to in silico Material Screening
• Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
• Neural fidelity warping for efficient robot morphology design
• MISO-wiLDCosts: Multi Information Source Optimization with Location Dependent Costs
• Discovering Diverse Athletic Jumping Strategies
• Multi-Fidelity Multi-Objective Bayesian Optimization: An Output Space Entropy Search Approach
• Lookahead Acquisition Functions for Finite-Horizon Time-Dependent Bayesian Optimization and Application to Quantum Optimal Control
• Efficient Hyperparameter Optimization for Physics-based Character Animation