4 papers
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Abbas Zaidi, Rina Friedberg +3
The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit u…
Evaluating Variance Estimates with Relative Efficiency
Kedar Karhadkar, Jack Klys, Daniel Ting +2
Experimentation platforms in industry must often deal with customer trust issues. Platforms must prove the validity of their claims as well as catch issues that arise. As a central…
Breaking the Winner's Curse with Bayesian Hybrid Shrinkage
Richard Mudd, Rina Friedberg, Ilya Gorbachev +2
A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings…
Counterfactual Evaluation of Ads Ranking Models through Domain Adaptation
Mohamed A. Radwan, Himaghna Bhattacharjee, Quinn Lanners +4
We propose a domain-adapted reward model that works alongside an Offline A/B testing system for evaluating ranking models. This approach effectively measures reward for ranking mod…