4 papers
Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification
Neeti Pokharna, Olivier Jeunen, Yatharth Saraf +1
Online evaluation of ranking and retrieval systems often relies on downstream monetization metrics such as app revenue or creator earnings. These metrics are typically heavy-tailed…
Powerful A/B-Testing Metrics and Where to Find Them
Olivier Jeunen, Shubham Baweja, Neeti Pokharna +1
Online controlled experiments, colloquially known as A/B-tests, are the bread and butter of real-world recommender system evaluation. Typically, end-users are randomly assigned som…
Variance Reduction in Ratio Metrics for Efficient Online Experiments
Shubham Baweja, Neeti Pokharna, Aleksei Ustimenko +1
Online controlled experiments, such as A/B-tests, are commonly used by modern tech companies to enable continuous system improvements. Despite their paramount importance, A/B-tests…
On Gradient Boosted Decision Trees and Neural Rankers: A Case-Study on Short-Video Recommendations at ShareChat
Olivier Jeunen, Hitesh Sagtani, Himanshu Doi +7
Practitioners who wish to build real-world applications that rely on ranking models, need to decide which modelling paradigm to follow. This is not an easy choice to make, as the r…