Generalizations of the Familywise Error Rate

Abstract

Consider the problem of simultaneously testing null hypotheses H_1,...,H_s. The usual approach to dealing with the multiplicity problem is to restrict attention to procedures that control the familywise error rate (FWER), the probability of even one false rejection. In many applications, particularly if s is large, one might be willing to tolerate more than one false rejection provided the number of such cases is controlled, thereby increasing the ability of the procedure to detect false null hypotheses. This suggests replacing control of the FWER by controlling the probability of k or more false rejections, which we call the k-FWER. We derive both single-step and stepdown procedures that control the k-FWER, without making any assumptions concerning the dependence structure of the p-values of the individual tests. In particular, we derive a stepdown procedure that is quite simple to apply, and prove that it cannot be improved without violation of control of the k-FWER. We also consider the false discovery proportion (FDP) defined by the number of false rejections divided by the total number of rejections (defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289-300] controls E(FDP). Here, we construct methods such that, for any γand α, P{FDP>γ}\leα. Two stepdown methods are proposed. The first holds under mild conditions on the dependence structure of p-values, while the second is more conservative but holds without any dependence assumptions.

Published at http://dx.doi.org/10.1214/009053605000000084 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (1)

• A stochastic process approach to false discovery control

Cited by in corpus (37)

• Microarrays, Empirical Bayes and the Two-Groups Model
• Control of the mean number of false discoveries, Bonferroni and stability of multiple testing
• Control of generalized error rates in multiple testing
• Multiple Testing and Error Control in Gaussian Graphical Model Selection
• Stepup procedures for control of generalizations of the familywise error rate
• A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing
• Comparing distributions by multiple testing across quantiles or CDF values
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Two simple sufficient conditions for FDR control
• A New Perspective on Robust $M$-Estimation: Finite Sample Theory and Applications to Dependence-Adjusted Multiple Testing
• Stepup procedures controlling generalized FWER and generalized FDR
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Power-enhanced multiple decision functions controlling family-wise error and false discovery rates
• Multiple testing via $FD{R}_L$ for large-scale imaging data
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Asymptotically optimal, sequential, multiple testing procedures with prior information on the number of signals
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• On a generalized false discovery rate
• Sequential multiple testing with generalized error control: an asymptotic optimality theory
• Exact calculations for false discovery proportion with application to least favorable configurations
• Generalizing Simes' test and Hochberg's stepup procedure
• Further results on controlling the false discovery proportion
• A new multiple testing method in the dependent case
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Type I error rate control for testing many hypotheses: a survey with proofs
• Optimal control of false discovery criteria in the two-group model
• Asymptotic properties of false discovery rate controlling procedures under independence
• Recent developments towards optimality in multiple hypothesis testing
• Kendall's tau in high-dimensional genomic parsimony
• Confidence-based Reasoning in Stochastic Constraint Programming
• Simultaneous critical values for $t$-tests in very high dimensions
• Strong approximations of level exceedences related to multiple hypothesis testing
• Graphical approaches for the control of generalised error rates
• FarmTest: Factor-Adjusted Robust Multiple Testing with Approximate False Discovery Control
• Regularizing stock return covariance matrices via multiple testing of correlations
• Empirical likelihood for the analysis of experimental designs
• Multiple testing of a function's monotonicity