Permutation-Based True Discovery Guarantee by Sum Tests

Abstract

Sum-based global tests are highly popular in multiple hypothesis testing. In this paper we propose a general closed testing procedure for sum tests, which provides lower confidence bounds for the proportion of true discoveries (TDP), simultaneously over all subsets of hypotheses. These simultaneous inferences come for free, i.e., without any adjustment of the alpha-level, whenever a global test is used. Our method allows for an exploratory approach, as simultaneity ensures control of the TDP even when the subset of interest is selected post hoc. It adapts to the unknown joint distribution of the data through permutation testing. Any sum test may be employed, depending on the desired power properties. We present an iterative shortcut for the closed testing procedure, based on the branch and bound algorithm, which converges to the full closed testing results, often after few iterations; even if it is stopped early, it controls the TDP. We compare the properties of different choices for the sum test through simulations, then we illustrate the feasibility of the method for high dimensional data on brain imaging and genomics data.

arXiv commentMain: 27 pages, 3 figures. Appendices: 19 pages, 7 figures

References in corpus (13)

• The Statistical Analysis of fMRI Data
• E-values: Calibration, combination, and applications
• Multiple Testing for Exploratory Research
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• An adaptively weighted statistic for detecting differential gene expression when combining multiple transcriptomic studies
• The sequential rejection principle of familywise error control
• Exact testing with random permutations
• Post hoc inference via joint family-wise error rate control
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Robust testing in generalized linear models by sign-flipping score contributions
• Notip: Non-parametric True Discovery Proportion control for brain imaging

Cited by in corpus (2)

• Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test
• The Lévy combination test