A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing

Abstract

In the spirit of modeling inference for microarrays as multiple testing for sparse mixtures, we present a similar approach to a simplified version of quantitative trait loci (QTL) mapping. Unlike in case of microarrays, where the number of tests usually reaches tens of thousands, the number of tests performed in scans for QTL usually does not exceed several hundreds. However, in typical cases, the sparsity $p$ of significant alternatives for QTL mapping is in the same range as for microarrays. For methodological interest, as well as some related applications, we also consider non-sparse mixtures. Using simulations as well as theoretical observations we study false discovery rate (FDR), power and misclassification probability for the Benjamini-Hochberg (BH) procedure and its modifications, as well as for various parametric and nonparametric Bayes and Parametric Empirical Bayes procedures. Our results confirm the observation of Genovese and Wasserman (2002) that for small p the misclassification error of BH is close to optimal in the sense of attaining the Bayes oracle. This property is shared by some of the considered Bayes testing rules, which in general perform better than BH for large or moderate $p$'s.

Published in at http://dx.doi.org/10.1214/193940307000000158 the IMS Collections (http://www.imstat.org/publications/imscollections.htm) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (3)

• Higher criticism for detecting sparse heterogeneous mixtures
• Generalizations of the Familywise Error Rate
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Cited by in corpus (21)

• Bayes and empirical-Bayes multiplicity adjustment in the variable-selection problem
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• The Horseshoe Estimator: Posterior Concentration around Nearly Black Vectors
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
• A nonparametric empirical Bayes framework for large-scale multiple testing
• Consistency of a recursive estimate of mixing distributions
• On false discovery rate thresholding for classification under sparsity
• Good, great, or lucky? Screening for firms with sustained superior performance using heavy-tailed priors
• Empirical priors and coverage of posterior credible sets in a sparse normal mean model
• Model Selection and Multiple Testing - A Bayesian and Empirical Bayes Overview and some New Results
• On nonparametric estimation of a mixing density via the predictive recursion algorithm
• A continuous multiple hypothesis testing framework for optimal exoplanet detection
• Benchmarking Historical Corporate Performance
• A New Step-down Procedure for Simultaneous Hypothesis Testing Under Dependence
• Bayesian Multiple Testing Under Sparsity for Polynomial-Tailed Distributions
• Asymptotic Minimaxity, Optimal Posterior Concentration and Asymptotic Bayes Optimality of Horseshoe-type Priors Under Sparsity
• Some Permutationllay Symmetric Multiple Hypotheses Testing Rules Under Dependent Set up
• Asymptotically optimal sequential FDR and pFDR control with (or without) prior information on the number of signals
• Flexible Bayesian Multiple Comparison Adjustment Using Dirichlet Process and Beta-Binomial Model Priors
• Epileptiform spikes in specific left temporal and mesial temporal structures disrupt verbal episodic memory encoding