Thompson Sampling with a Mixture Prior

Abstract

We study Thompson sampling (TS) in online decision making, where the uncertain environment is sampled from a mixture distribution. This is relevant in multi-task learning, where a learning agent faces different classes of problems. We incorporate this structure in a natural way by initializing TS with a mixture prior, and call the resulting algorithm MixTS. To analyze MixTS, we develop a novel and general proof technique for analyzing the concentration of mixture distributions. We use it to prove Bayes regret bounds for MixTS in both linear bandits and finite-horizon reinforcement learning. Our bounds capture the structure of the prior, depend on the number of mixture components and their widths. We also demonstrate the empirical effectiveness of MixTS in synthetic and real-world experiments.

arXiv commentProceedings of the 25th International Conference on Artificial Intelligence and Statistics

References in corpus (8)

• Analysis of Thompson Sampling for the multi-armed bandit problem
• Thompson Sampling for Contextual Bandits with Linear Payoffs
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Model-Based Reinforcement Learning with Value-Targeted Regression
• Model selection for contextual bandits
• Latent Contextual Bandits and their Application to Personalized Recommendations for New Users
• Information-Theoretic Confidence Bounds for Reinforcement Learning
• Learning to Optimize Via Posterior Sampling