8 papers
Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches
Pablo Alcain, Jason Kang, Mia Garrard +3
Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively…
Estimating the True Effect Size Distribution with SIMEX
Zhaoqi Li, Daniel Ting, Ilya Gorbachev +2
Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametr…
Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
Mohsen Malmir, Houssam Nassif, Danish Nasir Shaikh +2
Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Abbas Zaidi, Rina Friedberg +3
The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit u…
Evaluating Variance Estimates with Relative Efficiency
Kedar Karhadkar, Jack Klys, Daniel Ting +2
Experimentation platforms in industry must often deal with customer trust issues. Platforms must prove the validity of their claims as well as catch issues that arise. As a central…
Bayesian Predictive Probabilities for Online Experimentation
Abbas Zaidi, Rina Friedberg, Samir Khan +4
The widespread adoption of online randomized controlled experiments (A/B Tests) for decision-making has created ongoing capacity constraints which necessitate interim analyses. As…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Rina Friedberg, Ilya Gorbachev +2
A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings…
Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation
Mohamed A. Radwan, Himaghna Bhattacharjee, Quinn Lanners +4
We propose a domain-adapted reward model that works alongside an Offline A/B testing system for evaluating ranking models. This approach effectively measures reward for ranking mod…