Thompson Sampling via Local Uncertainty

Abstract

Thompson sampling is an efficient algorithm for sequential decision making, which exploits the posterior uncertainty to address the exploration-exploitation dilemma. There has been significant recent interest in integrating Bayesian neural networks into Thompson sampling. Most of these methods rely on global variable uncertainty for exploration. In this paper, we propose a new probabilistic modeling framework for Thompson sampling, where local latent variable uncertainty is used to sample the mean reward. Variational inference is used to approximate the posterior of the local variable, and semi-implicit structure is further introduced to enhance its expressiveness. Our experimental results on eight contextual bandit benchmark datasets show that Thompson sampling guided by local uncertainty achieves state-of-the-art performance while having low computational complexity.

arXiv commentICML 2020

References in corpus (16)

• Playing Atari with Deep Reinforcement Learning
• A Brief Survey of Deep Reinforcement Learning
• An Introduction to Deep Reinforcement Learning
• Weight Uncertainty in Neural Networks
• Analysis of Thompson Sampling for the multi-armed bandit problem
• Noisy Networks for Exploration
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• Parameter Space Noise for Exploration
• Variational Relevance Vector Machines
• Auxiliary Deep Generative Models
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Functional Variational Bayesian Neural Networks
• A Survey on Practical Applications of Multi-Armed and Contextual Bandits
• Variational Inference using Implicit Distributions
• Scalable Thompson Sampling via Optimal Transport