Higher criticism for detecting sparse heterogeneous mixtures

Abstract

Higher criticism, or second-level significance testing, is a multiple-comparisons concept mentioned in passing by Tukey. It concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis. Tukey suggested comparing the fraction of observed significances at a given α-level to the expected fraction under the joint null. In fact, he suggested standardizing the difference of the two quantities and forming a z-score; the resulting z-score tests the significance of the body of significance tests. We consider a generalization, where we maximize this z-score over a range of significance levels 0<α\leqα_0. We are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem: testing whether n normal means are all zero versus the alternative that a small fraction is nonzero. The subtlety of this ``sparse normal means'' testing problem can be seen from work of Ingster and Jin, who studied such problems in great detail. In their studies, they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p-values either for the bulk of the tests or for the few most highly significant tests. In this range, when the amplitude of nonzero means is calibrated with the fraction of nonzero means, the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses.

Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Statistics (http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/009053604000000265

Cited by in corpus (184)

• Planck 2013 results. XXV. Searches for cosmic strings and other topological defects
• On the Benjamini--Hochberg method
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery
• Optimal detection of sparse principal components in high dimension
• Two sample tests for high-dimensional covariance matrices
• The non-Gaussian Cold Spot in the 3-year WMAP data
• Estimating the proportion of false null hypotheses among a large number of independently tested hypotheses
• Global testing under sparse alternatives: ANOVA, multiple comparisons and the higher criticism
• Detection of an anomalous cluster in a network
• Group testing algorithms: bounds and simulations
• Innovated higher criticism for detecting sparse signals in correlated noise
• Goodness-of-fit tests via phi-divergences
• Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects
• Detection of a sparse submatrix of a high-dimensional noisy matrix
• Higher Criticism Statistic: Detecting and Identifying Non-Gaussianity in the WMAP First Year Data
• Genome-Wide Significance Levels and Weighted Hypothesis Testing
• On combinatorial testing problems
• On the false discovery rate and an asymptotically optimal rejection curve
• The Capacity of Adaptive Group Testing
• A comparison of the Benjamini-Hochberg procedure with some Bayesian rules for multiple testing
• Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models
• Asymptotic Bayes-optimality under sparsity of some multiple testing procedures
• Properties of higher criticism under strong dependence
• Estimation and confidence sets for sparse normal mixtures
• Minimax estimation of linear and quadratic functionals on sparsity classes
• Tests alternative to higher criticism for high-dimensional means under sparsity and column-wise dependence
• A comprehensive overview of the Cold Spot
• Simulation-Based Hypothesis Testing of High Dimensional Means Under Covariance Heterogeneity
• Computational Barriers to Estimation from Low-Degree Polynomials
• On the exact Berk-Jones statistics and their p-value calculation
• Post hoc inference via joint family-wise error rate control
• Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization
• Simultaneous Control of All False Discovery Proportions in Large-Scale Multiple Hypothesis Testing
• Cosmological non-Gaussian Signature Detection: Comparing Performance of Different Statistical Tests
• Optimal rates of convergence for estimating the null density and proportion of nonnull effects in large-scale multiple testing
• Feature selection by Higher Criticism thresholding: optimal phase diagram
• Higher criticism: $p$-values and criticism
• Hypothesis testing for high-dimensional sparse binary regression
• Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures
• Detection of correlations
• Signal Identification for Rare and Weak Features: Higher Criticism or False Discovery Rates?
• Weighted Hypothesis Testing
• Two-sample Hypothesis Testing for Inhomogeneous Random Graphs
• Only Closed Testing Procedures are Admissible for Controlling False Discovery Proportions
• Computational and Statistical Boundaries for Submatrix Localization in a Large Noisy Matrix
• Revisiting Marginal Regression
• Exact calculations for false discovery proportion with application to least favorable configurations
• Detection boundary and Higher Criticism approach for rare and weak genetic effects
• On the performance of FDR control: Constraints and a partial solution
• Optimal classification in sparse Gaussian graphic model
• A study of periodograms standardized using training data sets and application to exoplanet detection
• No Higher Criticism of the Bianchi Corrected WMAP Data
• Revisiting Multi-Subject Random Effects in fMRI: Advocating Prevalence Estimation
• On false discovery rate thresholding for classification under sparsity
• A Cramér moderate deviation theorem for Hotelling's $T^2$-statistic with applications to global tests
• 3D magneto-hydrodynamical simulations of stellar convective noise for improved exoplanet detection. I. Case of regularly sampled radial velocity observations
• Regularized Estimation and Testing for High-Dimensional Multi-Block Vector-Autoregressive Models
• Goodness of fit tests in terms of local levels with special emphasis on higher criticism tests
• Optimal detection of multi-sample aligned sparse signals
• Detecting positive correlations in a multivariate sample
• Adaptive testing on a regression function at a point
• False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments
• Fast calculation of boundary crossing probabilities for Poisson processes
• Better-Than-Chance Classification for Signal Detection
• Set-Based Tests for Genetic Association Using the Generalized Berk-Jones Statistic
• Nonparametric Empirical Bayes Estimation on Heterogeneous Data
• Higher Criticism to Compare Two Large Frequency Tables, with sensitivity to Possible Rare and Weak Differences
• Multi-sequence segmentation via score and higher-criticism tests
• Robust control of varying weak hyperspectral target detection with sparse non-negative representation
• Another Argument in Favour of Wilcoxon's Signed Rank Test
• On the Fundamental Limits of Adaptive Sensing
• Adaptive sensing performance lower bounds for sparse signal detection and support estimation
• Analysis of the Spatial Distribution of Galaxies by Multiscale Methods
• Rate optimal multiple testing procedure in high-dimensional regression
• Identifying Audio Adversarial Examples via Anomalous Pattern Detection
• Distribution-Free Tests for Sparse Heterogeneous Mixtures
• Robust Adaptive Rate-Optimal Testing for the White Noise Hypothesis
• Fast calculation of p-values for one-sided Kolmogorov-Smirnov type statistics
• Simultaneous Detection of Signal Regions Using Quadratic Scan Statistics With Applications in Whole Genome Association Studies
• Ensemble methods for testing a global null
• Phase Transitions for High Dimensional Clustering and Related Problems
• Detecting Rare and Weak Spikes in Large Covariance Matrices
• Searching for gluon quartic gauge couplings at muon colliders using the auto-encoder
• Detect anomalous quartic gauge couplings at muon colliders with quantum kernel k-means
• Estimation of False Discovery Proportion with Unknown Dependence
• Optimize the event selection strategy to study the anomalous quartic gauge couplings at muon colliders using the support vector machine and quantum support vector machine
• Influential Feature PCA for high dimensional clustering
• Incorporation of Sparsity Information in Large-scale Multiple Two-sample $t$ Tests
• Identifying the Support of Rectangular Signals in Gaussian Noise
• Distribution-free Multiple Testing
• Robust nearest-neighbor methods for classifying high-dimensional data
• Distance-based species tree estimation: information-theoretic trade-off between number of loci and sequence length under the coalescent
• Subset Scanning Over Neural Network Activations
• anomaly : Detection of Anomalous Structure in Time Series Data
• Higher Criticism for Discriminating Word-Frequency Tables and Testing Authorship
• Persistence Barcodes versus Kolmogorov Signatures: Detecting Modes of One-Dimensional Signals
• Testing High Dimensional Mean Under Sparsity
• Detection of sparse additive functions
• Estimating the number and effect sizes of non-null hypotheses
• Optimal Detection For Sparse Mixtures
• On the Non-asymptotic and Sharp Lower Tail Bounds of Random Variables
• Asymptotically Independent U-Statistics in High-Dimensional Testing
• A Practitioner's Guide to Multiple Testing Error Rates
• Non-asymptotic detection of two-component mixtures with unknown means
• Separating signal from noise
• Optimal Detection of Random Walks on Graphs: Performance Analysis via Statistical Physics
• Semi-supervised standardized detection of extrasolar planets
• Sequential Multiple Testing
• Covariate Assisted Variable Ranking
• Randomized tests for high-dimensional regression: A more efficient and powerful solution
• Sharp multiple testing boundary for sparse sequences
• A rate optimal procedure for sparse signal recovery under dependence
• On efficient estimators of the proportion of true null hypotheses in a multiple testing setup
• On the Grenander estimator at zero
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• An Impossibility Result for High Dimensional Supervised Learning
• On Minimax Exponents of Sparse Testing
• Unification of Rare and Weak Detection Models using Moderate Deviations Analysis and Log-Chisquared P-values
• Asymptotics for Markov chain mixture detection
• Variational Discriminant Analysis with Variable Selection
• Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares
• Active set algorithms for estimating shape-constrained density ratios
• The Sparse Variance Contamination Model
• Detection of Sparse Mixtures: Higher Criticism and Scan Statistic
• Search for anomalous quartic gauge couplings in the process ${\mu }^{+}{\mu }^{-}\to \bar{\nu }\nu \gamma \gamma$ with a nested local outlier factor
• Global testing against sparse alternatives in time-frequency analysis
• Lower bounds in multiple testing: A framework based on derandomized proxies
• Blockwise and coordinatewise thresholding to combine tests of different natures in modern ANOVA
• Estimation of the $l_2$-norm and testing in sparse linear regression with unknown variance
• Fundamental Limits of Exact Support Recovery in High Dimensions
• Detection of Correlations with Adaptive Sensing
• Variable Selection via Adaptive False Negative Control in Linear Regression
• Distributions and Statistical Power of Optimal Signal-Detection Methods In Finite Cases
• Analysis of error control in large scale two-stage multiple hypothesis testing
• Cross-phenotype meta-analysis reveals large-scale trans-eQTLs mediating patterns of transcriptional co-regulation
• eQTL Mapping via Effective SNP Ranking and Screening
• Multi-level Thresholding Test for High Dimensional Covariance Matrices
• Bayesian Models Applied to Cyber Security Anomaly Detection Problems
• Many Phenotypes without Many False Discoveries: Error Controlling Strategies for Multi-Traits Association Studies
• High Probability Lower Bounds for the Total Variation Distance
• Quantifying and Reducing Bias in Maximum Likelihood Estimation of Structured Anomalies
• Sparse Anomaly Detection Across Referentials: A Rank-Based Higher Criticism Approach
• Identification of Signal, Noise, and Indistinguishable Subsets in High-Dimensional Data Analysis
• Optimal Estimation of A Quadratic Functional and Detection of Simultaneous Signals
• FACT: Fast closed testing for exchangeable local tests
• Generalized Goodness-Of-Fit Tests for Correlated Data
• Higher Criticism Tuned Regression For Weak And Sparse Signals
• A Scan Procedure for Multiple Testing
• Two-sample testing of high-dimensional linear regression coefficients via complementary sketching
• A quantum machine learning classifier to search for new physics
• Optimality of the max test for detecting sparse signals with Gaussian or heavier tail
• Generalized k-Means in GLMs with Applications to the Outbreak of COVID-19 in the United States
• ANOMALYMAXQ:Anomaly-Structured Maximization to Query in Attributed Network
• Global Testing Against Sparse Alternatives under Ising Models
• Discussion of "Multiple Testing for Exploratory Research" by J. J. Goeman and A. Solari
• Significance testing for canonical correlation analysis in high dimensions
• Estimating the Null and the Proportion of non-Null effects in Large-scale Multiple Comparisons
• Noisy Hypotheses in the Age of Discovery Science
• Flexible Principal Component Analysis for Exponential Family Distributions
• Meta-analysis of mid-p-values: some new results based on the convex order
• Minimax rates for sparse signal detection under correlation
• A robust statistical method for Genome-wide association analysis of human copy number variation
• Optimal detection of weak positive latent dependence between two sequences of multiple tests
• Data volume and power of multiple tests with small sample size per null
• Detecting Sparse Heterogeneous Mixtures in a Two-Sample Problem
• Some detection tests for low complexity data models and unknown background distribution
• Sparse Recovery With Multiple Data Streams: A Sequential Adaptive Testing Approach
• Distributions associated with simultaneous multiple hypothesis testing
• The False Discovery Rate for Statistical Pattern Recognition
• Minimax rate of testing in sparse linear regression
• A Global Homogeneity Test for High-Dimensional Linear Regression
• Global testing under the sparse alternatives for single index models
• Higher criticism for rare and weak non-proportional hazard deviations in survival analysis
• Signal detection in extracellular neural ensemble recordings using higher criticism
• Diagonally-Dominant Principal Component Analysis
• Estimation of a Two-component Mixture Model with Applications to Multiple Testing
• Detecting Weak Distribution Shifts via Displacement Interpolation
• A grouped, selectively weighted false discovery rate procedure
• Heterocedasticity-Adjusted Ranking and Thresholding for Large-Scale Multiple Testing
• Subgroup Analysis for Longitudinal data via Semiparametric Additive Mixed Effect Model
• The Sparse Poisson Means Model
• The Forgetfulness of Balls and Bins
• On the Necessity of Irrelevant Variables
• Superconsistency of Tests in High Dimensions