Exact and asymptotically robust permutation tests

Abstract

Given independent samples from P and Q, two-sample permutation tests allow one to construct exact level tests when the null hypothesis is P=Q. On the other hand, when comparing or testing particular parameters $\theta$ of P and Q, such as their means or medians, permutation tests need not be level $\alpha$, or even approximately level $\alpha$ in large samples. Under very weak assumptions for comparing estimators, we provide a general test procedure whereby the asymptotic validity of the permutation test holds while retaining the exact rejection probability $\alpha$ in finite samples when the underlying distributions are identical. The ideas are broadly applicable and special attention is given to the k-sample problem of comparing general parameters, whereby a permutation test is constructed which is exact level $\alpha$ under the hypothesis of identical distributions, but has asymptotic rejection probability $\alpha$ under the more general null hypothesis of equality of parameters. A Monte Carlo simulation study is performed as well. A quite general theory is possible based on a coupling construction, as well as a key contiguity argument for the multinomial and multivariate hypergeometric distributions.

Published in at http://dx.doi.org/10.1214/13-AOS1090 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (29)

• Agnostic notes on regression adjustments to experimental data: Reexamining Freedman's critique
• Controlling the false discovery rate via knockoffs
• Randomization Tests for Weak Null Hypotheses in Randomized Experiments
• Cramér-type moderate deviations for Studentized two-sample $U$-statistics with applications
• Robust testing in generalized linear models by sign-flipping score contributions
• Selective inference with a randomized response
• Fast Approximation of Small p-values in Permutation Tests by Partitioning the Permutations
• Scalable and Efficient Hypothesis Testing with Random Forests
• Evaluating stochastic seeding strategies in networks
• An Assumption-Free Exact Test For Fixed-Design Linear Models With Exchangeable Errors
• Minimax optimality of permutation tests
• Covariate-adjusted Fisher randomization tests for the average treatment effect
• Inferring median survival differences in general factorial designs via permutation tests
• Beyond the Sharp Null: Randomization Inference, Bounded Null Hypotheses, and Confidence Intervals for Maximum Effects
• Robust Multivariate Nonparametric Tests via Projection-Averaging
• Randomization tests of copula symmetry
• Causal Inference: A Missing Data Perspective
• Measuring the Graph Concordance of Locally Dependent Observations
• Permutation-based tests for discontinuities in event studies
• Inference for the Mann-Whitney Effect for Right-Censored and Tied Data
• Randomization Inference beyond the Sharp Null: Bounded Null Hypotheses and Quantiles of Individual Treatment Effects
• A randomization-based perspective of analysis of variance: a test statistic robust to treatment effect heterogeneity
• An approximate randomization test for high-dimensional two-sample Behrens-Fisher problem under arbitrary covariances
• Permuting longitudinal data despite all the dependencies
• Fisher-Pitman permutation tests based on nonparametric Poisson mixtures with application to single cell genomics
• Permutation inference in factorial survival designs with the CASANOVA
• Resampling-based inference methods for comparing two coefficient alpha
• Prepivoted permutation tests
• Counting Defiers: Examples from Health Care