Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects

Abstract

In modern high-throughput data analysis, researchers perform a large number of statistical tests, expecting to find perhaps a small fraction of significant effects against a predominantly null background. Higher Criticism (HC) was introduced to determine whether there are any nonzero effects; more recently, it was applied to feature selection, where it provides a method for selecting useful predictive features from a large body of potentially useful features, among which only a rare few will prove truly useful. In this article, we review the basics of HC in both the testing and feature selection settings. HC is a flexible idea, which adapts easily to new situations; we point out simple adaptions to clique detection and bivariate outlier detection. HC, although still early in its development, is seeing increasing interest from practitioners; we illustrate this with worked examples. HC is computationally effective, which gives it a nice leverage in the increasingly more relevant "Big Data" settings we see today. We also review the underlying theoretical "ideology" behind HC. The Rare/Weak (RW) model is a theoretical framework simultaneously controlling the size and prevalence of useful/significant items among the useless/null bulk. The RW model shows that HC has important advantages over better known procedures such as False Discovery Rate (FDR) control and Family-wise Error control (FwER), in particular, certain optimality properties. We discuss the rare/weak phase diagram, a way to visualize clearly the class of RW settings where the true signals are so rare or so weak that detection and feature selection are simply impossible, and a way to understand the known optimality properties of HC.

Published at http://dx.doi.org/10.1214/14-STS506 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (15)

• Regularized estimation of large covariance matrices
• High-dimensional classification using features annealed independence rules
• The non-Gaussian Cold Spot in the 3-year WMAP data
• Goodness-of-fit tests via phi-divergences
• Genome-Wide Significance Levels and Weighted Hypothesis Testing
• A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• Higher criticism: $p$-values and criticism
• Detection boundary and Higher Criticism approach for rare and weak genetic effects
• No Higher Criticism of the Bianchi Corrected WMAP Data
• A Constrained L1 Minimization Approach to Sparse Precision Matrix Estimation
• A Cramér moderate deviation theorem for Hotelling's $T^2$-statistic with applications to global tests
• Robustness and accuracy of methods for high dimensional data analysis based on Student's t statistic
• Optimal Detection For Sparse Mixtures

Cited by in corpus (8)

• Identifying the Support of Rectangular Signals in Gaussian Noise
• Lower bounds in multiple testing: A framework based on derandomized proxies
• Subsampling Winner Algorithm for Feature Selection in Large Regression Data
• Analysis of error control in large scale two-stage multiple hypothesis testing
• Multi-level Thresholding Test for High Dimensional Covariance Matrices
• Optimality of the max test for detecting sparse signals with Gaussian or heavier tail
• Distributions and Statistical Power of Optimal Signal-Detection Methods In Finite Cases
• Signal detection in extracellular neural ensemble recordings using higher criticism