On the bias, risk and consistency of sample means in multi-armed bandits

Abstract

The sample mean is among the most well studied estimators in statistics, having many desirable properties such as unbiasedness and consistency. However, when analyzing data collected using a multi-armed bandit (MAB) experiment, the sample mean is biased and much remains to be understood about its properties. For example, when is it consistent, how large is its bias, and can we bound its mean squared error? This paper delivers a thorough and systematic treatment of the bias, risk and consistency of MAB sample means. Specifically, we identify four distinct sources of selection bias (sampling, stopping, choosing and rewinding) and analyze them both separately and together. We further demonstrate that a new notion of \emph{effective sample size} can be used to bound the risk of the sample mean under suitable loss functions. We present several carefully designed examples to provide intuition on the different sources of selection bias we study. Our treatment is nonparametric and algorithm-agnostic, meaning that it is not tied to a specific algorithm or goal. In a nutshell, our proofs combine variational representations of information-theoretic divergences with new martingale concentration inequalities.

48 pages

References in corpus (5)

• Time-uniform Chernoff bounds via nonnegative supermartingales
• Are sample means in multi-armed bandits positively or negatively biased?
• Why Adaptively Collected Data Have Negative Bias and How to Correct for It
• Mixture Martingales Revisited with Applications to Sequential Tests and Confidence Intervals
• KL-UCB-switch: optimal regret bounds for stochastic bandits from both a distribution-dependent and a distribution-free viewpoints

Cited by in corpus (7)

• Post-Contextual-Bandit Inference
• Weak Signal Asymptotics for Sequentially Randomized Experiments
• Debiasing Samples from Online Learning Using Bootstrap
• On conditional versus marginal bias in multi-armed bandits
• Doubly Robust Interval Estimation for Optimal Policy Evaluation in Online Learning
• A nonasymptotic law of iterated logarithm for general M-estimators
• Nonparametric iterated-logarithm extensions of the sequential generalized likelihood ratio test