Neural-BO: A Black-box Optimization Algorithm using Deep Neural Networks

Abstract

Bayesian Optimization (BO) is an effective approach for global optimization of black-box functions when function evaluations are expensive. Most prior works use Gaussian processes to model the black-box function, however, the use of kernels in Gaussian processes leads to two problems: first, the kernel-based methods scale poorly with the number of data points and second, kernel methods are usually not effective on complex structured high dimensional data due to curse of dimensionality. Therefore, we propose a novel black-box optimization algorithm where the black-box function is modeled using a neural network. Our algorithm does not need a Bayesian neural network to estimate predictive uncertainty and is therefore computationally favorable. We analyze the theoretical behavior of our algorithm in terms of regret bound using advances in NTK theory showing its efficient convergence. We perform experiments with both synthetic and real-world optimization tasks and show that our algorithm is more sample efficient compared to existing methods.

References in corpus (11)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Neural Contextual Bandits with Deep Representation and Shallow Exploration
• VerIDeep: Verifying Integrity of Deep Neural Networks through Sensitive-Sample Fingerprinting
• Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structure
• Exploiting correlation and budget constraints in Bayesian multi-armed bandit optimization

Cited by in corpus (1)

• Scalable neural network-based blackbox optimization