Sharp detection in PCA under correlations: all eigenvalues matter

Abstract

Principal component analysis (PCA) is a widely used method for dimension reduction. In high dimensional data, the "signal" eigenvalues corresponding to weak principal components (PCs) do not necessarily separate from the bulk of the "noise" eigenvalues. Therefore, popular tests based on the largest eigenvalue have little power to detect weak PCs. In the special case of the spiked model, certain tests asymptotically equivalent to linear spectral statistics (LSS)---averaging effects over all eigenvalues---were recently shown to achieve some power. We consider a nonparametric, non-Gaussian generalization of the spiked model to the setting of Marchenko and Pastur (1967). This allows a general bulk of the noise eigenvalues, accomodating correlated variables even under the null hypothesis of no significant PCs. We develop new tests based on LSS to detect weak PCs in this model. We show using the CLT for LSS that the optimal LSS satisfy a Fredholm integral equation of the first kind. We develop algorithms to solve it, building on our recent method for computing the limit empirical spectrum. In contrast to the standard spiked model, we find that under "widely spread" null eigenvalue distributions, the new tests have a lot of power.

46 pages, 9 figures

References in corpus (5)

• High Dimensional Statistical Inference and Random Matrices
• Central limit theorem for signal-to-interference ratio of reduced rank linear receiver
• Efficient Computation of Limit Spectra of Sample Covariance Matrices
• A Survey on the Eigenvalues Local Behavior of Large Complex Correlated Wishart Matrices
• Regularized LRT for Large Scale Covariance Matrices: One Sample Problem

Cited by in corpus (8)

• Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models
• Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization
• Fundamental limits of detection in the spiked Wigner model
• Optimal hypothesis testing for stochastic block models with growing degrees
• Finite Size Corrections and Likelihood Ratio Fluctuations in the Spiked Wigner Model
• Detection limits in the high-dimensional spiked rectangular model
• How to reduce dimension with PCA and random projections?
• Detecting Rare and Weak Spikes in Large Covariance Matrices