Combining Latent Space and Structured Kernels for Bayesian Optimization over Combinatorial Spaces

Abstract

We consider the problem of optimizing combinatorial spaces (e.g., sequences, trees, and graphs) using expensive black-box function evaluations. For example, optimizing molecules for drug design using physical lab experiments. Bayesian optimization (BO) is an efficient framework for solving such problems by intelligently selecting the inputs with high utility guided by a learned surrogate model. A recent BO approach for combinatorial spaces is through a reduction to BO over continuous spaces by learning a latent representation of structures using deep generative models (DGMs). The selected input from the continuous space is decoded into a discrete structure for performing function evaluation. However, the surrogate model over the latent space only uses the information learned by the DGM, which may not have the desired inductive bias to approximate the target black-box function. To overcome this drawback, this paper proposes a principled approach referred as LADDER. The key idea is to define a novel structure-coupled kernel that explicitly integrates the structural information from decoded structures with the learned latent space representation for better surrogate modeling. Our experiments on real-world benchmarks show that LADDER significantly improves over the BO over latent space method, and performs better or similar to state-of-the-art methods.

arXiv comment15 pages, 7 figures

References in corpus (21)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• A Tutorial on Bayesian Optimization
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Bayesian optimization for materials design
• Syntax-Directed Variational Autoencoder for Structured Data
• BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization
• Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization
• Generative Models for Automatic Chemical Design
• Bayesian Optimization of Combinatorial Structures
• Molecular Hypergraph Grammar with its Application to Molecular Optimization
• Combinatorial Bayesian Optimization using the Graph Cartesian Product
• Design by adaptive sampling
• Gaussian Process Molecule Property Prediction with FlowMO
• Improving black-box optimization in VAE latent space using decoder uncertainty
• BOSS: Bayesian Optimization over String Spaces
• Scalable Combinatorial Bayesian Optimization with Tractable Statistical models
• Bayesian Optimization with Approximate Set Kernels