Sparse Principal Components Analysis

Abstract

Principal components analysis (PCA) is a classical method for the reduction of dimensionality of data in the form of n observations (or cases) of a vector with p variables. For a simple model of factor analysis type, it is proved that ordinary PCA can produce a consistent (for n large) estimate of the principal factor if and only if p(n) is asymptotically of smaller order than n. There may be a basis in which typical signals have sparse representations: most co-ordinates have small signal energies. If such a basis (e.g. wavelets) is used to represent the signals, then the variation in many coordinates is likely to be small. Consequently, we study a simple "sparse PCA" algorithm: select a subset of coordinates of largest variance, estimate eigenvectors from PCA on the selected subset, threshold and reexpress in the original basis. We illustrate the algorithm on some exercise ECG data, and prove that in a single factor model, under an appropriate sparsity assumption, it yields consistent estimates of the principal factor.

This manuscript was written in late 2003; a much revised version is to appear, with discussion and later references, in the Journal of the American Statistical Association in 2009. The JASA revision cites this archive version for the proof of inconsistency of PCA

Cited by in corpus (33)

• Covariance regularization by thresholding
• Statistical physics of inference: Thresholds and algorithms
• PCA consistency in high dimension, low sample size context
• Finite sample approximation results for principal component analysis: a matrix perturbation approach
• High Dimensional Statistical Inference and Random Matrices
• Augmented sparse principal component analysis for high dimensional data
• High-dimensional covariance estimation by minimizing ${\ell }_1$-penalized log-determinant divergence
• Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization
• Sparse PCA via Covariance Thresholding
• SDCOR: Scalable Density-based Clustering for Local Outlier Detection in Massive-Scale Datasets
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Reducibility and Statistical-Computational Gaps from Secret Leakage
• Optimal Average-Case Reductions to Sparse PCA: From Weak Assumptions to Strong Hardness
• HyperPCA: a Powerful Tool to Extract Elemental Maps from Noisy Data Obtained in LIBS Mapping of Materials
• Subexponential-Time Algorithms for Sparse PCA
• On consistency and sparsity for sliced inverse regression in high dimensions
• Rank-one matrix estimation: analysis of algorithmic and information theoretic limits by the spatial coupling method
• 0-1 phase transitions in sparse spiked matrix estimation
• Random perturbation of low rank matrices: Improving classical bounds
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Supervised Learning for Multi-Block Incomplete Data
• A Well-Conditioned and Sparse Estimation of Covariance and Inverse Covariance Matrices Using a Joint Penalty
• Two-way kernel matrix puncturing: towards resource-efficient PCA and spectral clustering
• The Spectral Norm of Random Inner-Product Kernel Matrices
• Alternating direction method of multipliers for penalized zero-variance discriminant analysis
• Inconsistency of diagonal scaling under high-dimensional limit: a replica approach
• Machinery for Proving Sum-of-Squares Lower Bounds on Certification Problems
• A Compressed PCA Subspace Method for Anomaly Detection in High-Dimensional Data
• Stay on path: PCA along graph paths
• On the estimation of correlation in a binary sequence model
• Proximal Distance Algorithms: Theory and Examples
• Joint Beamforming and Power Control for Throughput Maximization in IRS-assisted MISO WPCNs
• Learning from Forecast Errors: A New Approach to Forecast Combinations