Always Valid Inference: Bringing Sequential Analysis to A/B Testing

Abstract

A/B tests are typically analyzed via frequentist p-values and confidence intervals; but these inferences are wholly unreliable if users endogenously choose samples sizes by *continuously monitoring* their tests. We define *always valid* p-values and confidence intervals that let users try to take advantage of data as fast as it becomes available, providing valid statistical inference whenever they make their decision. Always valid inference can be interpreted as a natural interface for a sequential hypothesis test, which empowers users to implement a modified test tailored to them. In particular, we show in an appropriate sense that the measures we develop tradeoff sample size and power efficiently, despite a lack of prior knowledge of the user's relative preference between these two goals. We also use always valid p-values to obtain multiple hypothesis testing control in the sequential context. Our methodology has been implemented in a large scale commercial A/B testing platform to analyze hundreds of thousands of experiments to date.

References in corpus (3)

• lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
• On the Complexity of A/B Testing
• Sequential Nonparametric Testing with the Law of the Iterated Logarithm

Cited by in corpus (19)

• Sequential estimation of quantiles with applications to A/B-testing and best-arm identification
• Admissible anytime-valid sequential inference must rely on nonnegative martingales
• Meta Dynamic Pricing: Transfer Learning Across Experiments
• Continuous Monitoring of A/B Tests without Pain: Optional Stopping in Bayesian Testing
• Concise Summarization of Heterogeneous Treatment Effect Using Total Variation Regularized Regression
• Non-Stationary Bandits with Habituation and Recovery Dynamics
• Comparing Sequential Forecasters
• A Bandit Approach to Multiple Testing with False Discovery Control
• SQR: Balancing Speed, Quality and Risk in Online Experiments
• Adaptive Experimentation with Delayed Binary Feedback
• Rapid Regression Detection in Software Deployments through Sequential Testing
• Near-Optimal Confidence Sequences for Bounded Random Variables
• A Decision Theoretic Approach to A/B Testing
• Bayesian Sequentially Monitored Multi-arm Experiments with Multiple Comparison Adjustments
• Analysis of Large Scale Web Experiments Using Sequences of Estimators
• Online Testing of Subgroup Treatment Effects Based on Value Difference
• Sparse Recovery With Multiple Data Streams: A Sequential Adaptive Testing Approach
• CONQ: CONtinuous Quantile Treatment Effects for Large-Scale Online Controlled Experiments
• Fundamental Limits of Testing the Independence of Irrelevant Alternatives in Discrete Choice