Randomization Tests for Weak Null Hypotheses in Randomized Experiments

Abstract

The Fisher randomization test (FRT) is appropriate for any test statistic, under a sharp null hypothesis that can recover all missing potential outcomes. However, it is often sought after to test a weak null hypothesis that the treatment does not affect the units on average. To use the FRT for a weak null hypothesis, we must address two issues. First, we need to impute the missing potential outcomes although the weak null hypothesis cannot determine all of them. Second, we need to choose a proper test statistic. For a general weak null hypothesis, we propose an approach to imputing missing potential outcomes under a compatible sharp null hypothesis. Building on this imputation scheme, we advocate a studentized statistic. The resulting FRT has multiple desirable features. First, it is model-free. Second, it is finite-sample exact under the sharp null hypothesis that we use to impute the potential outcomes. Third, it conservatively controls large-sample type I errors under the weak null hypothesis of interest. Therefore, our FRT is agnostic to the treatment effect heterogeneity. We establish a unified theory for general factorial experiments. We also extend it to stratified and clustered experiments.

References in corpus (10)

• Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique
• Exact and asymptotically robust permutation tests
• Lasso adjustments of treatment effect estimates in randomized experiments
• Sharp bounds on the variance in randomized experiments
• Regression adjustment in completely randomized experiments with a diverging number of covariates
• Multi-center clinical trials: Randomization and ancillary statistics
• Comments on the Neyman-Fisher Controversy and Its Consequences
• A studentized permutation test for three-arm trials in the 'gold standard' design
• A Unified Theory of Regression Adjustment for Design-based Inference
• Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects

Cited by in corpus (4)

• Sharp variance estimator and causal bootstrap in stratified randomized experiments
• Rejoinder to Reader Reaction "On exact randomization-based covariate-adjusted confidence intervals" by Jacob Fiksel
• On the term "randomization test"
• Hypothesis Testing in Sequentially Sampled Data: AdapRT to Maximize Power Beyond iid Sampling