The sequential rejection principle of familywise error control

Abstract

Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.

Published in at http://dx.doi.org/10.1214/10-AOS829 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Cited by in corpus (25)

• Multiple Testing for Exploratory Research
• Exact testing with random permutations
• Asymptotic optimality of the Westfall--Young permutation procedure for multiple testing under dependence
• Comparing three groups
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Type I error rate control for testing many hypotheses: a survey with proofs
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Optimal exact tests for multiple binary endpoints
• Sequential Selection Procedures and False Discovery Rate Control
• Permutation-Based True Discovery Guarantee by Sum Tests
• Spatial clustering of array CGH features in combination with hierarchical multiple testing
• On Selecting and Conditioning in Multiple Testing and Selective Inference
• Efficient and powerful familywise error control in genome-wide association studies using generalized linear models
• On Stepwise Control of Directional Errors under Independence and Some Dependence
• Post hoc false positive control for spatially structured hypotheses
• The Control of the False Discovery Rate in Fixed Sequence Multiple Testing
• Regularizing stock return covariance matrices via multiple testing of correlations
• Classes of Multiple Decision Functions Strongly Controlling FWER and FDR
• Asymptotic control of FWER under Gaussian assumption: application to correlation tests
• FACT: Fast closed testing for exchangeable local tests
• Online control of the familywise error rate
• Rejoinder to "Multiple Testing for Exploratory Research"
• Estimation and testing for multiple regulation of multivariate mixed outcomes
• Superconsistency of Tests in High Dimensions
• Efficient Multiple Testing Adjustment for Hierarchical Inference