Online learning in bandits with predicted context

Abstract

We consider the contextual bandit problem where at each time, the agent only has access to a noisy version of the context and the error variance (or an estimator of this variance). This setting is motivated by a wide range of applications where the true context for decision-making is unobserved, and only a prediction of the context by a potentially complex machine learning algorithm is available. When the context error is non-vanishing, classical bandit algorithms fail to achieve sublinear regret. We propose the first online algorithm in this setting with sublinear regret guarantees under mild conditions. The key idea is to extend the measurement error model in classical statistics to the online decision-making setting, which is nontrivial due to the policy being dependent on the noisy context observations. We further demonstrate the benefits of the proposed approach in simulation environments based on synthetic and real digital intervention datasets.

References in corpus (14)

• A Contextual-Bandit Approach to Personalized News Article Recommendation
• Sample Size Calculations for Micro-randomized Trials in mHealth
• Latent Contextual Bandits and their Application to Personalized Recommendations for New Users
• Personalized HeartSteps: A Reinforcement Learning Algorithm for Optimizing Physical Activity
• Stochastic Bandits with Context Distributions
• Power Constrained Bandits
• Post-Contextual-Bandit Inference
• Latent Bandits Revisited
• Online learning with Corrupted context: Corrupted Contextual Bandits
• Towards Agnostic Feature-based Dynamic Pricing: Linear Policies vs Linear Valuation with Unknown Noise
• Robust Stochastic Linear Contextual Bandits Under Adversarial Attacks
• Non-Stationary Latent Bandits
• A New Bandit Setting Balancing Information from State Evolution and Corrupted Context
• Distributed Stochastic Bandit Learning with Delayed Context Observation