Control of generalized error rates in multiple testing

Abstract

Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $\gamma$ and $\alpha$, $P\{FDP>\gamma \}\le \alpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Published in at http://dx.doi.org/10.1214/009053606000001622 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (6)

• Generalizations of the Familywise Error Rate
• A stochastic process approach to false discovery control
• Control of generalized error rates in multiple testing
• Stepup procedures for control of generalizations of the familywise error rate
• On stepdown control of the false discovery proportion
• On Optimality of Stepdown and Stepup Multiple Test Procedures

Cited by in corpus (19)

• Multiple Testing for Exploratory Research
• Control of generalized error rates in multiple testing
• Multiple Testing and Error Control in Gaussian Graphical Model Selection
• Stepup procedures for control of generalizations of the familywise error rate
• On a generalized false discovery rate
• Sequential multiple testing with generalized error control: an asymptotic optimality theory
• Recent Developments on Factor Models and its Applications in Econometric Learning
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Type I error rate control for testing many hypotheses: a survey with proofs
• Asymptotic properties of false discovery rate controlling procedures under independence
• Confidence and discoveries with e-values
• High-Dimensional Estimation, Basis Assets, and the Adaptive Multi-Factor Model
• A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
• Graphical approaches for the control of generalised error rates
• Sample size and positive false discovery rate control for multiple testing
• Regularizing stock return covariance matrices via multiple testing of correlations
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Semi-supervised multiple testing
• Compound p-Value Statistics for Multiple Testing Procedures