Taming Non-stationary Bandits: A Bayesian Approach

Abstract

We consider the multi armed bandit problem in non-stationary environments. Based on the Bayesian method, we propose a variant of Thompson Sampling which can be used in both rested and restless bandit scenarios. Applying discounting to the parameters of prior distribution, we describe a way to systematically reduce the effect of past observations. Further, we derive the exact expression for the probability of picking sub-optimal arms. By increasing the exploitative value of Bayes' samples, we also provide an optimistic version of the algorithm. Extensive empirical analysis is conducted under various scenarios to validate the utility of proposed algorithms. A comparison study with various state-of-the-arm algorithms is also included.

Submitted to NIPS 2017

References in corpus (1)

• Further Optimal Regret Bounds for Thompson Sampling

Cited by in corpus (7)

• Adapting Behaviour for Learning Progress
• Nonstationary Stochastic Multiarmed Bandits: UCB Policies and Minimax Regret
• AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning
• Boltzmann Exploration Expectation-Maximisation
• Distribution-dependent and Time-uniform Bounds for Piecewise i.i.d Bandits
• On Limited-Memory Subsampling Strategies for Bandits
• Odds-Ratio Thompson Sampling to Control for Time-Varying Effect