Simultaneous inference: When should hypothesis testing problems be combined?

Abstract

Modern statisticians are often presented with hundreds or thousands of hypothesis testing problems to evaluate at the same time, generated from new scientific technologies such as microarrays, medical and satellite imaging devices, or flow cytometry counters. The relevant statistical literature tends to begin with the tacit assumption that a single combined analysis, for instance, a False Discovery Rate assessment, should be applied to the entire set of problems at hand. This can be a dangerous assumption, as the examples in the paper show, leading to overly conservative or overly liberal conclusions within any particular subclass of the cases. A simple Bayesian theory yields a succinct description of the effects of separation or combination on false discovery rate analyses. The theory allows efficient testing within small subclasses, and has applications to ``enrichment,'' the detection of multi-case effects.

Published in at http://dx.doi.org/10.1214/07-AOAS141 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (3)

• On testing the significance of sets of genes
• Size, power and false discovery rates
• Random-set methods identify distinct aspects of the enrichment signal in gene-set analysis

Cited by in corpus (11)

• When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing
• Genome-Wide Significance Levels and Weighted Hypothesis Testing
• The Future of Indirect Evidence
• Covariate powered cross-weighted multiple testing
• Power-enhanced multiple decision functions controlling family-wise error and false discovery rates
• Beyond the E-value: stratified statistics for protein domain prediction
• Empirical null and false discovery rate inference for exponential families
• Measuring support for a hypothesis about a random parameter without estimating its unknown prior
• Rejoinder: Microarrays, Empirical Bayes and the Two-Groups Model
• Classes of Multiple Decision Functions Strongly Controlling FWER and FDR
• Multiscale Comparative Connectomics