4 papers · 1 filter
Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches
Pablo Alcain, Jason Kang, Mia Garrard +3
Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively…
Estimating the True Effect Size Distribution with SIMEX
Zhaoqi Li, Daniel Ting, Ilya Gorbachev +2
Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametr…
Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
Mohsen Malmir, Houssam Nassif, Danish Nasir Shaikh +2
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Abbas Zaidi, Rina Friedberg +3
The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit u…