Innovated higher criticism for detecting sparse signals in correlated noise

Abstract

Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.

Published in at http://dx.doi.org/10.1214/09-AOS764 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (12)

• Regularized estimation of large covariance matrices
• The non-Gaussian Cold Spot in the 3-year WMAP data
• Goodness-of-fit tests via phi-divergences
• Robustness of multiple testing procedures against dependence
• On false discovery control under dependence
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• Asymptotic minimaxity of False Discovery Rate thresholding for sparse exponential data
• Feature selection by Higher Criticism thresholding: optimal phase diagram
• A new multiple testing method in the dependent case
• Asymptotic inference for high-dimensional data
• Long-range correlation in the whole human genome

Cited by in corpus (27)

• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Detection of an anomalous cluster in a network
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Innovated interaction screening for high-dimensional nonlinear classification
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence
• UPS delivers optimal phase diagram in high-dimensional variable selection
• Asymptotic optimality of the Westfall--Young permutation procedure for multiple testing under dependence
• Hypothesis testing for high-dimensional sparse binary regression
• Detection of correlations
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Detection boundary and Higher Criticism approach for rare and weak genetic effects
• Optimal classification in sparse Gaussian graphic model
• A study of periodograms standardized using training data sets and application to exoplanet detection
• A Cramér moderate deviation theorem for Hotelling's $T^2$-statistic with applications to global tests
• Optimal detection of multi-sample aligned sparse signals
• Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests
• Detecting positive correlations in a multivariate sample
• Robust control of varying weak hyperspectral target detection with sparse non-negative representation
• Localization of Matrix Factorizations
• Fast calculation of p-values for one-sided Kolmogorov-Smirnov type statistics
• Ensemble methods for testing a global null
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Global testing against sparse alternatives in time-frequency analysis
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach
• Superconsistency of Tests in High Dimensions
• Higher criticism for rare and weak non-proportional hazard deviations in survival analysis