Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses

Abstract

We consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses, focusing on the situation in which the proportion of false null hypotheses is very small. We propose a family of methods for establishing lower $100(1-\alpha )$ confidence bounds for this proportion, based on the empirical distribution of the $p$-values of the tests. Methods in this family are then compared in terms of ability to consistently estimate the proportion by letting $\alpha \to 0$ as the number of hypothesis tests increases and the proportion decreases. This work is motivated by a signal detection problem that occurs in astronomy.

Published at http://dx.doi.org/10.1214/009053605000000741 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• Higher criticism for detecting sparse heterogeneous mixtures
• A stochastic process approach to false discovery control

Cited by in corpus (53)

• Innovated higher criticism for detecting sparse signals in correlated noise
• Goodness-of-fit tests via phi-divergences
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• On the false discovery rate and an asymptotically optimal rejection curve
• On false discovery control under dependence
• A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• An empirical Bayes mixture method for effect size and false discovery rate estimation
• A New Perspective on Robust $M$-Estimation: Finite Sample Theory and Applications to Dependence-Adjusted Multiple Testing
• Stepup procedures controlling generalized FWER and generalized FDR
• Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing
• Asymptotic optimality of the Westfall--Young permutation procedure for multiple testing under dependence
• Higher criticism: $p$-values and criticism
• Adaptive FDR control under independence and dependence
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings
• On a generalized false discovery rate
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Revisiting Marginal Regression
• On the performance of FDR control: Constraints and a partial solution
• On approximate validation of models: A Kolmogorov-Smirnov based approach
• Adaptive testing on a regression function at a point
• Recent developments towards optimality in multiple hypothesis testing
• False discovery rate control with multivariate $p$-values
• Nonparametric Empirical Bayes Estimation on Heterogeneous Data
• Adaptive sensing performance lower bounds for sparse signal detection and support estimation
• Fast calculation of p-values for one-sided Kolmogorov-Smirnov type statistics
• Uniformly consistently estimating the proportion of false null hypotheses via Lebesgue-Stieltjes integral equations
• Simultaneous critical values for $t$-tests in very high dimensions
• Sample size and positive false discovery rate control for multiple testing
• FarmTest: Factor-Adjusted Robust Multiple Testing with Approximate False Discovery Control
• A generalized Fourier approach to estimating the null parameters and proportion of non-null effects in large-scale multiple testing
• Estimating the number and effect sizes of non-null hypotheses
• On efficient estimators of the proportion of true null hypotheses in a multiple testing setup
• Box-constrained monotone $L_{\infty }$-approximations to Lipschitz regularizations, with applications to robust testing
• Variable Selection via Adaptive False Negative Control in Linear Regression
• Higher Criticism Tuned Regression For Weak And Sparse Signals
• False Discovery Rate Computation: Illustrations and Modifications
• Quantifying and Reducing Bias in Maximum Likelihood Estimation of Structured Anomalies
• High Probability Lower Bounds for the Total Variation Distance
• Two-component Mixture Model in the Presence of Covariates
• Straight to the Source: Detecting Aggregate Objects in Astronomical Images with Proper Error Control
• On the future of astrostatistics: statistical foundations and statistical practice
• Sparse Recovery With Multiple Data Streams: A Sequential Adaptive Testing Approach
• Estimation of a Two-component Mixture Model with Applications to Multiple Testing
• Discussion of "Multiple Testing for Exploratory Research" by J. J. Goeman and A. Solari
• On the consistency of adaptive multiple tests
• On the Necessity of Irrelevant Variables
• The Sparse Poisson Means Model
• Extending the Patra-Sen Approach to Estimating the Background Component in a Two-Component Mixture Model
• A grouped, selectively weighted false discovery rate procedure
• A New Step-down Procedure for Simultaneous Hypothesis Testing Under Dependence