Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing

Abstract

An important estimation problem that is closely related to large-scale multiple testing is that of estimating the null density and the proportion of nonnull effects. A few estimators have been introduced in the literature; however, several important problems, including the evaluation of the minimax rate of convergence and the construction of rate-optimal estimators, remain open. In this paper, we consider optimal estimation of the null density and the proportion of nonnull effects. Both minimax lower and upper bounds are derived. The lower bound is established by a two-point testing argument, where at the core is the novel construction of two least favorable marginal densities $f_1$ and $f_2$. The density $f_1$ is heavy tailed both in the spatial and frequency domains and $f_2$ is a perturbation of $f_1$ such that the characteristic functions associated with $f_1$ and $f_2$ match each other in low frequencies. The minimax upper bound is obtained by constructing estimators which rely on the empirical characteristic function and Fourier analysis. The estimator is shown to be minimax rate optimal. Compared to existing methods in the literature, the proposed procedure not only provides more precise estimates of the null density and the proportion of the nonnull effects, but also yields more accurate results when used inside some multiple testing procedures which aim at controlling the False Discovery Rate (FDR). The procedure is easy to implement and numerical results are given.

Published in at http://dx.doi.org/10.1214/09-AOS696 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (7)

• Microarrays, Empirical Bayes and the Two-Groups Model
• On the false discovery rate and an asymptotically optimal rejection curve
• Estimation and confidence sets for sparse normal mixtures
• Asymptotic minimaxity of False Discovery Rate thresholding for sparse exponential data
• Adaptive FDR control under independence and dependence
• Asymptotic properties of false discovery rate controlling procedures under independence
• A leave-p-out based estimation of the proportion of null hypotheses

Cited by in corpus (6)

• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Asymptotically minimax empirical Bayes estimation of a sparse normal mean vector
• Deconvolution for an atomic distribution: rates of convergence
• Uniformly consistently estimating the proportion of false null hypotheses via Lebesgue-Stieltjes integral equations
• Post hoc false discovery proportion inference under a Hidden Markov Model