Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects

Abstract

In modern high-throughput data analysis, researchers perform a large number of statistical tests, expecting to find perhaps a small fraction of significant effects against a predominantly null background. Higher Criticism (HC) was introduced to determine whether there are any nonzero effects; more recently, it was applied to feature selection, where it provides a method for selecting useful predictive features from a large body of potentially useful features, among which only a rare few will prove truly useful. In this article, we review the basics of HC in both the testing and feature selection settings. HC is a flexible idea, which adapts easily to new situations; we point out simple adaptions to clique detection and bivariate outlier detection. HC, although still early in its development, is seeing increasing interest from practitioners; we illustrate this with worked examples. HC is computationally effective, which gives it a nice leverage in the increasingly more relevant "Big Data" settings we see today. We also review the underlying theoretical "ideology" behind HC. The Rare/Weak (RW) model is a theoretical framework simultaneously controlling the size and prevalence of useful/significant items among the useless/null bulk. The RW model shows that HC has important advantages over better known procedures such as False Discovery Rate (FDR) control and Family-wise Error control (FwER), in particular, certain optimality properties. We discuss the rare/weak phase diagram, a way to visualize clearly the class of RW settings where the true signals are so rare or so weak that detection and feature selection are simply impossible, and a way to understand the known optimality properties of HC.

arXiv commentPublished at http://dx.doi.org/10.1214/14-STS506 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (41)

• High-dimensional graphs and variable selection with the Lasso
• Regularized estimation of large covariance matrices
• Higher criticism for detecting sparse heterogeneous mixtures
• High-dimensional classification using features annealed independence rules
• The non-Gaussian Cold Spot in the 3-year WMAP data
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Detection of an anomalous cluster in a network
• Innovated higher criticism for detecting sparse signals in correlated noise
• Goodness-of-fit tests via phi-divergences
• Detection of a sparse submatrix of a high-dimensional noisy matrix
• Higher Criticism Statistic: Detecting and Identifying Non-Gaussianity in the WMAP First Year Data
• Feature selection in omics prediction problems using cat scores and false nondiscovery rate control
• Genome-Wide Significance Levels and Weighted Hypothesis Testing
• On combinatorial testing problems
• A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence
• A comprehensive overview of the Cold Spot
• UPS delivers optimal phase diagram in high-dimensional variable selection
• Cosmological non-Gaussian Signature Detection: Comparing Performance of Different Statistical Tests
• Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing
• Feature selection by Higher Criticism thresholding: optimal phase diagram
• Covariate assisted screening and estimation
• Higher criticism: $p$-values and criticism
• Signal Identification for Rare and Weak Features: Higher Criticism or False Discovery Rates?
• Detection boundary and Higher Criticism approach for rare and weak genetic effects
• Optimal classification in sparse Gaussian graphic model
• No Higher Criticism of the Bianchi Corrected WMAP Data
• A Cramér moderate deviation theorem for Hotelling's $T^2$-statistic with applications to global tests
• A Constrained L1 Minimization Approach to Sparse Precision Matrix Estimation
• Distribution-Free Tests for Sparse Heterogeneous Mixtures
• Robustness and accuracy of methods for high dimensional data analysis based on Student's t statistic
• Cosmological models discrimination with Weak Lensing
• Influential Feature PCA for high dimensional clustering
• Detection of sparse additive functions
• Reanalysis of F-statistic gravitational-wave searches with the higher criticism statistic
• Optimal Detection For Sparse Mixtures
• An Impossibility Result for High Dimensional Supervised Learning

Cited by in corpus (31)

• Statistically-Robust Clustering Techniques for Mapping Spatial Hotspots: A Survey
• Two-sample Hypothesis Testing for Inhomogeneous Random Graphs
• Higher Criticism to Compare Two Large Frequency Tables, with sensitivity to Possible Rare and Weak Differences
• Robust control of varying weak hyperspectral target detection with sparse non-negative representation
• Which bridge estimator is optimal for variable selection?
• Permutation-Based True Discovery Guarantee by Sum Tests
• Recent Advances in Text Analysis
• Phase Transitions for High Dimensional Clustering and Related Problems
• Ensemble methods for testing a global null
• Detecting Rare and Weak Spikes in Large Covariance Matrices
• Influential Feature PCA for high dimensional clustering
• Sharp Computational-Statistical Phase Transitions via Oracle Computational Model
• Identifying the Support of Rectangular Signals in Gaussian Noise
• Interactive Martingale Tests for the Global Null
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Asymptotically Independent U-Statistics in High-Dimensional Testing
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Structure in Galaxy Distribution. III. Fourier Transforming the Universe
• Subsampling Winner Algorithm for Feature Selection in Large Regression Data
• Lower bounds in multiple testing: A framework based on derandomized proxies
• Unification of Rare and Weak Detection Models using Moderate Deviations Analysis and Log-Chisquared P-values
• Distributions and Statistical Power of Optimal Signal-Detection Methods In Finite Cases
• Analysis of error control in large scale two-stage multiple hypothesis testing
• Optimality of the max test for detecting sparse signals with Gaussian or heavier tail
• High Probability Lower Bounds for the Total Variation Distance
• Multi-level Thresholding Test for High Dimensional Covariance Matrices
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach
• Higher criticism for rare and weak non-proportional hazard deviations in survival analysis
• Significance testing for canonical correlation analysis in high dimensions
• Signal detection in extracellular neural ensemble recordings using higher criticism
• Detecting Weak Distribution Shifts via Displacement Interpolation