Optimal Detection For Sparse Mixtures

Abstract

Detection of sparse signals arises in a wide range of modern scientific studies. The focus so far has been mainly on Gaussian mixture models. In this paper, we consider the detection problem under a general sparse mixture model and obtain an explicit expression for the detection boundary. It is shown that the fundamental limits of detection is governed by the behavior of the log-likelihood ratio evaluated at an appropriate quantile of the null distribution. We also establish the adaptive optimality of the higher criticism procedure across all sparse mixtures satisfying certain mild regularity conditions. In particular, the general results obtained in this paper recover and extend in a unified manner the previously known results on sparse detection far beyond the conventional Gaussian model and other exponential families.

arXiv commentsubmitted to IEEE Transactions on Information Theory

References in corpus (6)

• Higher criticism for detecting sparse heterogeneous mixtures
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Searching for a trail of evidence in a maze
• Higher Criticism Statistic: Detecting and Identifying Non-Gaussianity in the WMAP First Year Data
• Universal and Composite Hypothesis Testing via Mismatched Divergence
• Cosmological non-Gaussian Signature Detection: Comparing Performance of Different Statistical Tests

Cited by in corpus (2)

• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence