Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly

Abstract

Bayesian Optimisation (BO) refers to a suite of techniques for global optimisation of expensive black box functions, which use introspective Bayesian models of the function to efficiently search for the optimum. While BO has been applied successfully in many applications, modern optimisation tasks usher in new challenges where conventional methods fail spectacularly. In this work, we present Dragonfly, an open source Python library for scalable and robust BO. Dragonfly incorporates multiple recently developed methods that allow BO to be applied in challenging real world settings; these include better methods for handling higher dimensional domains, methods for handling multi-fidelity evaluations when cheap approximations of an expensive function are available, methods for optimising over structured combinatorial spaces, such as the space of neural network architectures, and methods for handling parallel evaluations. Additionally, we develop new methodological improvements in BO for selecting the Bayesian model, selecting the acquisition function, and optimising over complex domains with different variable types and additional constraints. We compare Dragonfly to a suite of other packages and algorithms for global optimisation and demonstrate that when the above methods are integrated, they enable significant improvements in the performance of BO. The Dragonfly library is available at dragonfly.github.io.

arXiv commentJournal of Machine Learning Research 2020, Special Issue on Bayesian Optimization

References in corpus (31)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Cosmological Constraints from the SDSS Luminous Red Galaxies
• DARTS: Differentiable Architecture Search
• Scrutinizing Exotic Cosmological Models Using ESSENCE Supernova Data Combined with Other Cosmological Probes
• Scalable Bayesian Optimization Using Deep Neural Networks
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• Random Search and Reproducibility for Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Batch Bayesian Optimization via Local Penalization
• Additive Gaussian Processes
• A Bayesian model selection analysis of WMAP3
• Freeze-Thaw Bayesian Optimization
• Batched Large-scale Bayesian Optimization in High-dimensional Spaces
• Multi-fidelity Bayesian Optimisation with Continuous Approximations
• A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes
• Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions
• Active Learning from Weak and Strong Labelers
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• Black-box optimization of noisy functions with unknown smoothness
• Constrained Bayesian Optimization for Automatic Chemical Design
• Bayesian Optimization for Synthetic Gene Design
• An Entropy Search Portfolio for Bayesian Optimization
• High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groups
• A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations
• Joint Optimization and Variable Selection of High-dimensional Gaussian Processes
• Additive Approximations in High Dimensional Nonparametric Regression via the SALSA
• Calculating Luminosity Distance versus Redshift in FLRW Cosmology via Homotopy Perturbation Method
• The Multi-fidelity Multi-armed Bandit
• ProBO: Versatile Bayesian Optimization Using Any Probabilistic Programming Language
• Noisy Blackbox Optimization with Multi-Fidelity Queries: A Tree Search Approach

Cited by in corpus (12)

• Autonomous optimization of nonaqueous battery electrolytes via robotic experimentation and machine learning
• Autonomous discovery of battery electrolytes with robotic experimentation and machine-learning
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalarizations
• Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning
• Efficient Neural Interaction Function Search for Collaborative Filtering
• Stability and Generalization of Bilevel Programming in Hyperparameter Optimization
• Practitioner Motives to Use Different Hyperparameter Optimization Methods
• HyperZero: A Customized End-to-End Auto-Tuning System for Recommendation with Hourly Feedback
• Meta-Learning to Improve Pre-Training
• Learning Minimax Estimators via Online Learning