High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling

Abstract

We introduce a novel semi-supervised learning approach, named Teacher-Student Bayesian Optimization ($\text{TSBO}$), integrating the teacher-student paradigm into BO to minimize expensive labeled data queries for the first time. $\text{TSBO}$ incorporates a teacher model, an unlabeled data sampler, and a student model. The student is trained on unlabeled data locations generated by the sampler, with pseudo labels predicted by the teacher. The interplay between these three components implements a unique selective regularization to the teacher in the form of student feedback. This scheme enables the teacher to predict high-quality pseudo labels, enhancing the generalization of the GP surrogate model in the search space. To fully exploit $\text{TSBO}$, we propose two optimized unlabeled data samplers to construct effective student feedback that well aligns with the objective of Bayesian optimization. Furthermore, we quantify and leverage the uncertainty of the teacher-student model for the provision of reliable feedback to the teacher in the presence of risky pseudo-label predictions. $\text{TSBO}$ demonstrates significantly improved sample-efficiency in several global optimization tasks under tight labeled data budgets.

arXiv comment15 pages

References in corpus (22)

• Auto-Encoding Variational Bayes
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Unsupervised Data Augmentation for Consistency Training
• Temporal Ensembling for Semi-Supervised Learning
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning
• Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• USB: A Unified Semi-supervised Learning Benchmark for Classification
• High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces
• All SMILES Variational Autoencoder
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• Re-Examining Linear Embeddings for High-Dimensional Bayesian Optimization
• Improving black-box optimization in VAE latent space using decoder uncertainty
• Local Latent Space Bayesian Optimization over Structured Inputs
• Increasing the Scope as You Learn: Adaptive Bayesian Optimization in Nested Subspaces
• Semi-supervised Embedding Learning for High-dimensional Bayesian Optimization
• Global Adversarial Attacks for Assessing Deep Learning Robustness
• PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidance