Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces

Abstract

In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional parameter spaces that explicitly includes information about which parameters are relevant in a given structure. We show that this kernel improves model quality and Bayesian optimization results over several simpler baseline kernels.

6 pages, 3 figures. Appeared in the NIPS 2013 workshop on Bayesian optimization

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• Slice sampling covariance hyperparameters of latent Gaussian models
• A Kernel for Hierarchical Parameter Spaces

Cited by in corpus (18)

• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Automated Machine Learning: State-of-The-Art and Open Challenges
• A Framework to Integrate Mode Choice in the Design of Mobility-on-Demand Systems
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Bayesian Optimization of Combinatorial Structures
• AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• How Powerful are Performance Predictors in Neural Architecture Search?
• Learnable Embedding Space for Efficient Neural Architecture Compression
• Hybrid Parameter Search and Dynamic Model Selection for Mixed-Variable Bayesian Optimization
• Additive Tree-Structured Conditional Parameter Spaces in Bayesian Optimization: A Novel Covariance Function and a Fast Implementation
• Neural Architecture Search Over a Graph Search Space
• Transfer NAS: Knowledge Transfer between Search Spaces with Transformer Agents
• Deep-n-Cheap: An Automated Search Framework for Low Complexity Deep Learning
• Large Scale Neural Architecture Search with Polyharmonic Splines
• Funneled Bayesian Optimization for Design, Tuning and Control of Autonomous Systems