A Linear Bandit for Seasonal Environments

Abstract

Contextual bandit algorithms are extremely popular and widely used in recommendation systems to provide online personalised recommendations. A recurrent assumption is the stationarity of the reward function, which is rather unrealistic in most of the real-world applications. In the music recommendation scenario for instance, people's music taste can abruptly change during certain events, such as Halloween or Christmas, and revert to the previous music taste soon after. We would therefore need an algorithm which can promptly react to these changes. Moreover, we would like to leverage already observed rewards collected during different stationary periods which can potentially reoccur, without the need of restarting the learning process from scratch. A growing literature has addressed the problem of reward's non-stationarity, providing algorithms that could quickly adapt to the changing environment. However, up to our knowledge, there is no algorithm which deals with seasonal changes of the reward function. Here we present a contextual bandit algorithm which detects and adapts to abrupt changes of the reward function and leverages previous estimations whenever the environment falls back to a previously observed state. We show that the proposed method can outperform state-of-the-art algorithms for non-stationary environments. We ran our experiment on both synthetic and real datasets.

References in corpus (11)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms
• Efficient Contextual Bandits in Non-stationary Worlds
• Learning Contextual Bandits in a Non-stationary Environment
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• Nearly Optimal Adaptive Procedure with Change Detection for Piecewise-Stationary Bandit
• A New Algorithm for Non-stationary Contextual Bandits: Efficient, Optimal, and Parameter-free
• Thompson Sampling in Switching Environments with Bayesian Online Change Point Detection
• Hedging the Drift: Learning to Optimize under Non-Stationarity
• A Simple Approach for Non-stationary Linear Bandits
• Sequential Monte Carlo Bandits

Cited by in corpus (2)

• Learning and Optimization with Seasonal Patterns
• Lifelong Learning in Multi-Armed Bandits