4 papers · 1 filter
On the Benefit of Blocking for Online Experiments with Skewed Data
Abbas Zaidi, Matthew Reimherr, Rina Friedberg +2
This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%…
Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches
Pablo Alcain, Jason Kang, Mia Garrard +3
Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively…
Bayesian Predictive Probabilities for Online Experimentation
Abbas Zaidi, Rina Friedberg, Samir Khan +4
The widespread adoption of online randomized controlled experiments (A/B Tests) for decision-making has created ongoing capacity constraints which necessitate interim analyses. As…
Treatment Effect Learning Under Sequential Randomization
Rina Friedberg, Richard Mudd, Patrick Johnstone +4
Sequential treatment assignments in online experiments lead to complex dependency structures, often rendering identification, estimation and inference over treatments a challenge.…