Estimation and confidence sets for sparse normal mixtures

Abstract

For high dimensional statistical models, researchers have begun to focus on situations which can be described as having relatively few moderately large coefficients. Such situations lead to some very subtle statistical problems. In particular, Ingster and Donoho and Jin have considered a sparse normal means testing problem, in which they described the precise demarcation or detection boundary. Meinshausen and Rice have shown that it is even possible to estimate consistently the fraction of nonzero coordinates on a subset of the detectable region, but leave unanswered the question of exactly in which parts of the detectable region consistent estimation is possible. In the present paper we develop a new approach for estimating the fraction of nonzero means for problems where the nonzero means are moderately large. We show that the detection region described by Ingster and Donoho and Jin turns out to be the region where it is possible to consistently estimate the expected fraction of nonzero coordinates. This theory is developed further and minimax rates of convergence are derived. A procedure is constructed which attains the optimal rate of convergence in this setting. Furthermore, the procedure also provides an honest lower bound for confidence intervals while minimizing the expected length of such an interval. Simulations are used to enable comparison with the work of Meinshausen and Rice, where a procedure is given but where rates of convergence have not been discussed. Extensions to more general Gaussian mixture models are also given.

Published in at http://dx.doi.org/10.1214/009053607000000334 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (5)

• Higher criticism for detecting sparse heterogeneous mixtures
• A stochastic process approach to false discovery control
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• An adaptation theory for nonparametric confidence intervals
• Estimation and confidence sets for sparse normal mixtures

Cited by in corpus (23)

• Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences
• Hypothesis test for normal mixture models: The EM approach
• Innovated higher criticism for detecting sparse signals in correlated noise
• Goodness-of-fit tests via phi-divergences
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• An empirical Bayes mixture method for effect size and false discovery rate estimation
• Feature selection by Higher Criticism thresholding: optimal phase diagram
• Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing
• Signal Identification for Rare and Weak Features: Higher Criticism or False Discovery Rates?
• Optimal classification in sparse Gaussian graphic model
• Revisiting Multi-Subject Random Effects in fMRI: Advocating Prevalence Estimation
• Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests
• Adaptive testing on a regression function at a point
• Adaptive sensing performance lower bounds for sparse signal detection and support estimation
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Comment: Microarrays, Empirical Bayes and the Two-Group Model
• Non-asymptotic detection of two-component mixtures with unknown means
• Global testing against sparse alternatives in time-frequency analysis
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach