Bayesian Optimization with Approximate Set Kernels

Abstract

We propose a practical Bayesian optimization method over sets, to minimize a black-box function that takes a set as a single input. Because set inputs are permutation-invariant, traditional Gaussian process-based Bayesian optimization strategies which assume vector inputs can fall short. To address this, we develop a Bayesian optimization method with \emph{set kernel} that is used to build surrogate functions. This kernel accumulates similarity over set elements to enforce permutation-invariance, but this comes at a greater computational cost. To reduce this burden, we propose two key components: (i) a more efficient approximate set kernel which is still positive-definite and is an unbiased estimator of the true set kernel with upper-bounded variance in terms of the number of subsamples, (ii) a constrained acquisition function optimization over sets, which uses symmetry of the feasible region that defines a set input. Finally, we present several numerical experiments which demonstrate that our method outperforms other methods.

arXiv comment18 pages, 7 figures, 5 tables, accepted for publication in Machine Learning Journal

References in corpus (12)

• Scikit-learn: Machine Learning in Python
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Open3D: A Modern Library for 3D Data Processing
• A Tutorial on Bayesian Optimization
• Bayesian optimization for materials design
• Bayesian Multi-Scale Optimistic Optimization
• Bayesian Optimization for Synthetic Gene Design
• Poisson Random Fields for Dynamic Feature Models

Cited by in corpus (4)

• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Kernels over Sets of Finite Sets using RKHS Embeddings, with Application to Bayesian (Combinatorial) Optimization
• Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces