4 papers
Testing for trends in online experiments
Chris Haulk, Lee Richardson, Jacopo Soriano
Time-varying treatment effects can be estimated with linear regression paired with jackknife confidence intervals. In experiments at a large online video platform, the test based o…
Evaluating for the long term: Learnings from industry
Leif Sigerson, Tom Cunningham, Winston Chou +22
Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and shar…
Choosing a Proxy Metric from Past Experiments
Nilesh Tripuraneni, Lee Richardson, Alexander D'Amour +2
In many randomized experiments, the treatment effect of the long-term metric (i.e. the primary outcome of interest) is often difficult or infeasible to measure. Such long-term metr…
Pareto optimal proxy metrics
Alessandro Zito, Dylan Greaves, Jacopo Soriano +1
North star metrics and online experimentation play a central role in how technology companies improve their products. In many practical settings, however, evaluating experiments ba…