8 papers
Halve-Then-Greedy: Adaptive Sampling for Posterior Success with Near-Optimal Instance Dependence
Yinan Li, Kwang-Sung Jun, Kapilan Balagopalan +4
We study fixed-budget adaptive experimentation aimed at maximizing the probability of obtaining sufficient posterior evidence that a recommended design's mean performance exceeds a…
On the Benefit of Blocking for Online Experiments with Skewed Data
Abbas Zaidi, Matthew Reimherr, Rina Friedberg +2
This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%…
Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches
Pablo Alcain, Jason Kang, Mia Garrard +3
Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively…
Incentive Alignment in Online Experimentation
Ermis Soumalias, {Ermis Soumalias, Richard Mudd +1
Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimizatio…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Abbas Zaidi, Rina Friedberg +3
The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit u…
Bayesian Predictive Probabilities for Online Experimentation
Abbas Zaidi, Rina Friedberg, Samir Khan +4
The widespread adoption of online randomized controlled experiments (A/B Tests) for decision-making has created ongoing capacity constraints which necessitate interim analyses. As…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Rina Friedberg, Ilya Gorbachev +2
A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings…
Treatment Effect Learning Under Sequential Randomization
Rina Friedberg, Richard Mudd, Patrick Johnstone +4
Sequential treatment assignments in online experiments lead to complex dependency structures, often rendering identification, estimation and inference over treatments a challenge.…