On Post-Selection Inference in A/B Tests

Abstract

When interpreting A/B tests, we typically focus only on the statistically significant results and take them by face value. This practice, termed post-selection inference in the statistical literature, may negatively affect both point estimation and uncertainty quantification, and therefore hinder trustworthy decision making in A/B testing. To address this issue, in this paper we explore two seemingly unrelated paths, one based on supervised machine learning and the other on empirical Bayes, and propose post-selection inferential approaches that combine the strengths of both. Through large-scale simulated and empirical examples, we demonstrate that our proposed methodologies stand out among other existing ones in both reducing post-selection biases and improving confidence interval coverage rates, and discuss how they can be conveniently adjusted to real-life scenarios.

References in corpus (5)

• Shrinkage Estimators in Online Experiments
• Nonparametric Empirical Bayes Estimation on Heterogeneous Data
• Concise Summarization of Heterogeneous Treatment Effect Using Total Variation Regularized Regression
• Applying the Delta method in metric analytics: A practical guide with novel ideas
• A Decision Theoretic Approach to A/B Testing

Cited by in corpus (1)

• Empirical Bayes Selection for Value Maximization