What is a randomization test?

Abstract

The meaning of randomization tests has become obscure in statistics education and practice over the last century. This article makes a fresh attempt at rectifying this core concept of statistics. A new term -- "quasi-randomization test" -- is introduced to define significance tests based on theoretical models and distinguish these tests from the "randomization tests" based on the physical act of randomization. The practical importance of this distinction is illustrated through a real stepped-wedge cluster-randomized trial. Building on the recent literature of randomization inference, a general framework of conditional randomization tests is developed and some practical methods to construct conditioning events are given. The proposed terminology and framework are then applied to understand several widely used (quasi-)randomization tests, including Fisher's exact test, permutation tests for treatment effect, quasi-randomization tests for independence and conditional independence, adaptive randomization, and conformal prediction.

45 pages, 2 figures. Accepted for publication in the Journal of American Statistical Association on 26th March, 2023. arXiv admin note: substantial text overlap with arXiv:2104.10618

References in corpus (5)

• Rerandomization to improve covariate balance in experiments
• Causal Inference in Genetic Trio Studies
• Multi-center clinical trials: Randomization and ancillary statistics
• Comments on the Neyman-Fisher Controversy and Its Consequences
• A Two-Sample Conditional Distribution Test Using Conformal Prediction and Weighted Rank Sum

Cited by in corpus (3)

• Rejoinder to Reader Reaction "On exact randomization-based covariate-adjusted confidence intervals" by Jacob Fiksel
• On the term "randomization test"
• Sample Observed Effects: Enumeration, Randomization and Generalization