Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions

Abstract

We consider the class of all multiple testing methods controlling tail probabilities of the false discovery proportion, either for one random set or simultaneously for many such sets. This class encompasses methods controlling familywise error rate, generalized familywise error rate, false discovery exceedance, joint error rate, simultaneous control of all false discovery proportions, and others, as well as seemingly unrelated methods such as gene set testing in genomics and cluster inference methods in neuroimaging. We show that all such methods are either equivalent to a closed testing method, or are uniformly improved by one. Moreover, we show that a closed testing method is admissible as a method controlling tail probabilities of false discovery proportions if and only if all its local tests are admissible. This implies that, when designing such methods, it is sufficient to restrict attention to closed testing methods only. We demonstrate the practical usefulness of this design principle by constructing a uniform improvement of a recently proposed method.

References in corpus (16)

• Higher criticism for detecting sparse heterogeneous mixtures
• Valid post-selection inference
• Generalizations of the Familywise Error Rate
• A stochastic process approach to false discovery control
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• Multiple Testing for Exploratory Research
• Innovated higher criticism for detecting sparse signals in correlated noise
• The sequential rejection principle of familywise error control
• Post hoc inference via joint family-wise error rate control
• Stepup procedures controlling generalized FWER and generalized FDR
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Further results on controlling the false discovery proportion
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests

Cited by in corpus (10)

• Comparing three groups
• Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test
• Confidence and discoveries with e-values
• Permutation-Based True Discovery Guarantee by Sum Tests
• On Selecting and Conditioning in Multiple Testing and Selective Inference
• Optimal multiple testing and design in clinical trials
• An exhaustive ADDIS principle for online FWER control
• FDP control in multivariate linear models using the bootstrap
• The Online Closure Principle
• Simultaneous directional inference