Multiple Testing for Exploratory Research

Abstract

Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.

Published in at http://dx.doi.org/10.1214/11-STS356 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• Control of generalized error rates in multiple testing
• The sequential rejection principle of familywise error control

Cited by in corpus (23)

• E-values: Calibration, combination, and applications
• Exact testing with random permutations
• A Tutorial on Statistically Sound Pattern Discovery
• Post hoc inference via joint family-wise error rate control
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• A shortcut for Hommel's procedure in linearithmic time
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Post-selection Inference in Multiverse Analysis (PIMA): an inferential framework based on the sign flipping score test
• Confidence and discoveries with e-values
• Notip: Non-parametric True Discovery Proportion control for brain imaging
• Permutation-Based True Discovery Guarantee by Sum Tests
• Post-selection inference for e-value based confidence intervals
• Discussion of "Multiple Testing for Exploratory Research" by J. J. Goeman and A. Solari
• Controlling Costs: Feature Selection on a Budget
• On Selecting and Conditioning in Multiple Testing and Selective Inference
• Regularizing stock return covariance matrices via multiple testing of correlations
• FDP control in multivariate linear models using the bootstrap
• Post hoc false discovery proportion inference under a Hidden Markov Model
• The Online Closure Principle
• Simultaneous directional inference