Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values

Abstract

Many testing problems are readily amenable to randomised tests such as those employing data splitting. However despite their usefulness in principle, randomised tests have obvious drawbacks. Firstly, two analyses of the same dataset may lead to different results. Secondly, the test typically loses power because it does not fully utilise the entire sample. As a remedy to these drawbacks, we study how to combine the test statistics or p-values resulting from multiple random realisations such as through random data splits. We develop rank-transformed subsampling as a general method for delivering large sample inference about the combined statistic or p-value under mild assumptions. We apply our methodology to a wide range of problems, including testing unimodality in high-dimensional data, testing goodness-of-fit of parametric quantile regression models, testing no direct effect in a sequentially randomised trial and calibrating cross-fit double machine learning confidence intervals. In contrast to existing p-value aggregation schemes that can be highly conservative, our method enjoys type-I error control that asymptotically approaches the nominal level. Moreover, compared to using the ordinary subsampling, we show that our rank transform can remove the first-order bias in approximating the null under alternatives and greatly improve power.

83 pages; typo correction and bibliography updates

References in corpus (15)

• ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R
• Variable selection with error control: Another look at Stability Selection
• To Cluster, or Not to Cluster: An Analysis of Clusterability Methods
• E-values: Calibration, combination, and applications
• A general framework for inference on algorithm-agnostic variable importance
• Statistical Significance of Clustering using Soft Thresholding
• Nested Markov Properties for Acyclic Directed Mixed Graphs
• Goodness of fit tests for high-dimensional linear models
• Semiparametric doubly robust targeted double machine learning: a review
• Significance tests of feature relevance for a black-box learner
• A lack-of-fit test for quantile regression models with high-dimensional covariates
• Multiple-Splitting Projection Test for High-Dimensional Mean Vectors
• The Projected Covariance Measure for assumption-lean variable significance testing
• A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond
• Nonlinear Regression with Residuals: Causal Estimation with Time-varying Treatments and Covariates

Cited by in corpus (3)

• Combining exchangeable p-values
• Variable importance measures for heterogeneous treatment effects
• Algorithm-agnostic significance testing in supervised learning with multimodal data