Epsilon-Greedy Thompson Sampling to Bayesian Optimization

Abstract

Bayesian optimization (BO) has become a powerful tool for solving simulation-based engineering optimization problems thanks to its ability to integrate physical and mathematical understandings, consider uncertainty, and address the exploitation-exploration dilemma. Thompson sampling (TS) is a preferred solution for BO to handle the exploitation-exploration trade-off. While it prioritizes exploration by generating and minimizing random sample paths from probabilistic models -- a fundamental ingredient of BO -- TS weakly manages exploitation by gathering information about the true objective function after it obtains new observations. In this work, we improve the exploitation of TS by incorporating the $\epsilon$-greedy policy, a well-established selection strategy in reinforcement learning. We first delineate two extremes of TS, namely the generic TS and the sample-average TS. The former promotes exploration, while the latter favors exploitation. We then adopt the $\epsilon$-greedy policy to randomly switch between these two extremes. Small and large values of $\epsilon$ govern exploitation and exploration, respectively. By minimizing two benchmark functions and solving an inverse problem of a steel cantilever beam, we empirically show that $\epsilon$-greedy TS equipped with an appropriate $\epsilon$ is more robust than its two extremes, matching or outperforming the better of the generic TS and the sample-average TS.

References in corpus (9)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• GPflow: A Gaussian process library using TensorFlow
• Max-value Entropy Search for Efficient Bayesian Optimization
• Greed is Good: Exploration and Exploitation Trade-offs in Bayesian Optimisation
• Bayesian Optimization with Output-Weighted Optimal Sampling
• Multi-Objective Bayesian Optimization for Accelerator Tuning
• Multi-fidelity Machine Learning for Uncertainty Quantification and Optimization
• Accelerating Approximate Thompson Sampling with Underdamped Langevin Monte Carlo

Cited by in corpus (2)

• Digital Twins and Civil Engineering Phases: Reorienting Adoption Strategies
• Multi-fidelity Machine Learning for Uncertainty Quantification and Optimization