Permutation-based simultaneous confidence bounds for the false discovery proportion

Abstract

When multiple hypotheses are tested, interest is often in ensuring that the proportion of false discoveries (FDP) is small with high confidence. In this paper, confidence upper bounds for the FDP are constructed, which are simultaneous over all rejection cut-offs. In particular this allows the user to select a set of hypotheses post hoc such that the FDP lies below some constant with high confidence. Our method uses permutations to account for the dependence structure in the data. So far only Meinshausen provided an exact, permutation-based and computationally feasible method for simultaneous FDP bounds. We provide an exact method, which uniformly improves this procedure. Further, we provide a generalization of this method. It lets the user select the shape of the simultaneous confidence bounds. This gives the user more freedom in determining the power properties of the method. Interestingly, several existing permutation methods, such as Significance Analysis of Microarrays (SAM) and Westfall and Young's maxT method, are obtained as special cases.

References in corpus (8)

• In silico prediction of protein-protein interactions in human macrophages
• A stochastic process approach to false discovery control
• Multiple Testing for Exploratory Research
• Exact testing with random permutations
• Two simple sufficient conditions for FDR control
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Asymptotic optimality of the Westfall--Young permutation procedure for multiple testing under dependence
• Further results on controlling the false discovery proportion

Cited by in corpus (11)

• Post hoc inference via joint family-wise error rate control
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Robust testing in generalized linear models by sign-flipping score contributions
• Another look at the Lady Tasting Tea and differences between permutation tests and randomization tests
• Confidence and discoveries with e-values
• Permutation-Based True Discovery Guarantee by Sum Tests
• FDP control in multivariate linear models using the bootstrap
• Semi-supervised multiple testing
• On agnostic post hoc approaches to false positive control
• On the term "randomization test"