Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimization

Abstract

Bayesian optimization (BO) conventionally relies on handcrafted acquisition functions (AFs) to sequentially determine the sample points. However, it has been widely observed in practice that the best-performing AF in terms of regret can vary significantly under different types of black-box functions. It has remained a challenge to design one AF that can attain the best performance over a wide variety of black-box functions. This paper aims to attack this challenge through the perspective of reinforced few-shot AF learning (FSAF). Specifically, we first connect the notion of AFs with Q-functions and view a deep Q-network (DQN) as a surrogate differentiable AF. While it serves as a natural idea to combine DQN and an existing few-shot learning method, we identify that such a direct combination does not perform well due to severe overfitting, which is particularly critical in BO due to the need of a versatile sampling policy. To address this, we present a Bayesian variant of DQN with the following three features: (i) It learns a distribution of Q-networks as AFs based on the Kullback-Leibler regularization framework. This inherently provides the uncertainty required in sampling for BO and mitigates overfitting. (ii) For the prior of the Bayesian DQN, we propose to use a demo policy induced by an off-the-shelf AF for better training stability. (iii) On the meta-level, we leverage the meta-loss of Bayesian model-agnostic meta-learning, which serves as a natural companion to the proposed FSAF. Moreover, with the proper design of the Q-networks, FSAF is general-purpose in that it is agnostic to the dimension and the cardinality of the input domain. Through extensive experiments, we demonstrate that the FSAF achieves comparable or better regrets than the state-of-the-art benchmarks on a wide variety of synthetic and real-world test functions.

arXiv comment21 pages, 8 figures

References in corpus (19)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Theoretical Models of Learning to Learn
• A Simple Neural Attentive Meta-Learner
• Reinforcement Learning with Deep Energy-Based Policies
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• A Tutorial on Bayesian Optimization
• Bayesian optimization for materials design
• Improving Object Detection with Deep Convolutional Networks via Bayesian Optimization and Structured Prediction
• Max-value Entropy Search for Efficient Bayesian Optimization
• Fast Context Adaptation via Meta-Learning
• Stein Variational Policy Gradient
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• GLASSES: Relieving The Myopia Of Bayesian Optimisation
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Q-Learning in enormous action spaces via amortized approximate maximization
• Variational Deep Q Network