BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization

Abstract

Bayesian optimization provides sample-efficient global optimization for a broad range of applications, including automatic machine learning, engineering, physics, and experimental design. We introduce BoTorch, a modern programming framework for Bayesian optimization that combines Monte-Carlo (MC) acquisition functions, a novel sample average approximation optimization approach, auto-differentiation, and variance reduction techniques. BoTorch's modular design facilitates flexible specification and optimization of probabilistic models written in PyTorch, simplifying implementation of new acquisition functions. Our approach is backed by novel theoretical convergence results and made practical by a distinctive algorithmic foundation that leverages fast predictive distributions, hardware acceleration, and deterministic optimization. We also propose a novel "one-shot" formulation of the Knowledge Gradient, enabled by a combination of our theoretical and software contributions. In experiments, we demonstrate the improved sample efficiency of BoTorch relative to other popular libraries.

Cited by in corpus (47)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
• Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation
• Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
• Latent Map Gaussian Processes for Mixed Variable Metamodeling
• Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement
• Empirica: a virtual lab for high-throughput macro-level experiments
• Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
• Centimeter-scale nanomechanical resonators with low dissipation
• Deep unfitted Nitsche method for elliptic interface problems
• Bayesian Optimisation for Sequential Experimental Design with Applications in Additive Manufacturing
• Local policy search with Bayesian optimization
• High-dimensional Automated Radiation Therapy Treatment Planning via Bayesian Optimization
• Adaptive Nonparametric Psychophysics
• Are we Forgetting about Compositional Optimisers in Bayesian Optimisation?
• Multi-Objective Bayesian Optimization over High-Dimensional Search Spaces
• High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
• Better call Surrogates: A hybrid Evolutionary Algorithm for Hyperparameter optimization
• Bayesian Optimization with High-Dimensional Outputs
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• GIBBON: General-purpose Information-Based Bayesian OptimisatioN
• Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum
• Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
• Multi-Step Budgeted Bayesian Optimization with Unknown Evaluation Costs
• Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces
• Kernel Interpolation for Scalable Online Gaussian Processes
• Benchmarking Structured Policies and Policy Optimization for Real-World Dexterous Object Manipulation
• Unsupervised Reservoir Computing for Solving Ordinary Differential Equations
• A Pragmatic Look at Deep Imitation Learning
• Bayesian Optimisation for Constrained Problems
• Self-supervised learning for fast and scalable time series hyper-parameter tuning
• Bayesian Optimization for Ion Beam Centroid Correction
• Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Bayesian Optimization of Sampling Densities in MRI
• Surrogate-guided optimization in quantum networks
• MBORE: Multi-objective Bayesian Optimisation by Density-Ratio Estimation
• Quasi-Newton Quasi-Monte Carlo for variational Bayes
• Towards Robust and Automatic Hyper-Parameter Tunning
• Advancing the Pareto front for thin-film materials using a self-driving laboratory
• JUMBO: Scalable Multi-task Bayesian Optimization using Offline Data
• Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks
• High-Dimensional Bayesian Optimization via Random Projection of Manifold Subspaces
• Choice functions based multi-objective Bayesian optimisation
• Knowledge Gradient for Multi-Objective Bayesian Optimization with Decoupled Evaluations
• Optimizing High-Dimensional Physics Simulations via Composite Bayesian Optimization