Sparse principal component analysis and iterative thresholding

Abstract

Principal component analysis (PCA) is a classical dimension reduction method which projects data onto the principal subspace spanned by the leading eigenvectors of the covariance matrix. However, it behaves poorly when the number of features p is comparable to, or even much larger than, the sample size n. In this paper, we propose a new iterative thresholding approach for estimating principal subspaces in the setting where the leading eigenvectors are sparse. Under a spiked covariance model, we find that the new approach recovers the principal subspace and leading eigenvectors consistently, and even optimally, in a range of high-dimensional sparse settings. Simulated examples also demonstrate its competitive performance.

Published in at http://dx.doi.org/10.1214/13-AOS1097 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (5)

• Truncated Power Method for Sparse Eigenvalue Problems
• Finite sample approximation results for principal component analysis: a matrix perturbation approach
• High-dimensional analysis of semidefinite relaxations for sparse principal components
• Augmented sparse principal component analysis for high dimensional data
• Consistency of Sparse PCA in High Dimension, Low Sample Size Contexts

Cited by in corpus (107)

• Regularized estimation of large covariance matrices
• Sparse PCA: Optimal rates and adaptive estimation
• Optimal detection of sparse principal components in high dimension
• Truncated Power Method for Sparse Eigenvalue Problems
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• Projected principal component analysis in factor models
• Minimax sparse principal subspace estimation in high dimensions
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models
• Computational Lower Bounds for Sparse PCA
• Statistical and computational trade-offs in estimation of sparse principal components
• A Tight Bound of Hard Thresholding
• Achieving Optimal Misclassification Proportion in Stochastic Block Model
• Sparse CCA: Adaptive Estimation and Computational Barriers
• Orthogonal Sparse PCA and Covariance Estimation via Procrustes Reformulation
• Near-Optimal Stochastic Approximation for Online Principal Component Estimation
• Optimality and Sub-optimality of PCA for Spiked Random Matrices and Synchronization
• Sum-of-Squares Lower Bounds for Sparse PCA
• Sparse CCA via Precision Adjusted Iterative Thresholding
• Do semidefinite relaxations solve sparse PCA up to the information limit?
• Nonconvex Statistical Optimization: Minimax-Optimal Sparse PCA in Polynomial Time
• Minimax estimation in sparse canonical correlation analysis
• Eigenvectors from Eigenvalues Sparse Principal Component Analysis (EESPCA)
• Sparse and Functional Principal Components Analysis
• Sparsistency and agnostic inference in sparse PCA
• Asymptotics of Empirical Eigen-structure for Ultra-high Dimensional Spiked Covariance Model
• Large Covariance Estimation through Elliptical Factor Models
• Rate Optimal Denoising of Simultaneously Sparse and Low Rank Matrices
• Optimal linear estimation under unknown nonlinear transform
• Multivariate Analysis of Nonparametric Estimates of Large Correlation Matrices
• Rate-optimal posterior contraction for sparse PCA
• Sparse PCA with Oracle Property
• A Sparse SVD Method for High-dimensional Data
• Optimal Average-Case Reductions to Sparse PCA: From Weak Assumptions to Strong Hardness
• Convergence of eigenvector empirical spectral distribution of sample covariance matrices
• Expandable Factor Analysis
• An ${\ell }_{\infty }$ Eigenvector Perturbation Bound and Its Application to Robust Covariance Estimation
• Sparse PCA via Bipartite Matchings
• Convergence rates of eigenvector empirical spectral distribution of large dimensional sample covariance matrix
• Penalized Orthogonal Iteration for Sparse Estimation of Generalized Eigenvalue Problem
• A generalization of regularized dual averaging and its dynamics
• The estimation error of general first order methods
• Robust Covariance and Scatter Matrix Estimation under Huber's Contamination Model
• Principal Components and Regularized Estimation of Factor Models
• Estimation of functionals of sparse covariance matrices
• Exploration of Large Networks with Covariates via Fast and Universal Latent Space Model Fitting
• Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow
• Non-negative Principal Component Analysis: Message Passing Algorithms and Sharp Asymptotics
• An Overview on the Estimation of Large Covariance and Precision Matrices
• Recovery of simultaneous low rank and two-way sparse coefficient matrices, a nonconvex approach
• Selective Factor Extraction in High Dimensions
• Instability, Computational Efficiency and Statistical Accuracy
• Optimal Estimation and Rank Detection for Sparse Spiked Covariance Matrices
• Partially Observed Dynamic Tensor Response Regression
• Influential Feature PCA for high dimensional clustering
• Towards Sample-Optimal Compressive Phase Retrieval with Sparse and Generative Priors
• Estimation of False Discovery Proportion with Unknown Dependence
• Large scale analysis of generalization error in learning using margin based classification methods
• Power Iteration for Tensor PCA
• Hessian Eigenspectra of More Realistic Nonlinear Models
• Subspace Estimation from Unbalanced and Incomplete Data Matrices: ${\ell }_{2,\infty }$ Statistical Guarantees
• Linear spectral statistics of eigenvectors of anisotropic sample covariance matrices
• Optimal Structured Principal Subspace Estimation: Metric Entropy and Minimax Rates
• Scalable Interpretable Multi-Response Regression via SEED
• Provable Sample-Efficient Sparse Phase Retrieval Initialized by Truncated Power Method
• Adversarially Robust Low Dimensional Representations
• Likelihood Ratio Test in Multivariate Linear Regression: from Low to High Dimension
• Online Learning for Sparse PCA in High Dimensions: Exact Dynamics and Phase Transitions
• Robust Covariance Estimation for Approximate Factor Models
• Stochastic Primal-Dual Method on Riemannian Manifolds with Bounded Sectional Curvature
• Sparse Generalized Eigenvalue Problem: Optimal Statistical Rates via Truncated Rayleigh Flow
• The Benefit of Group Sparsity in Group Inference with De-biased Scaled Group Lasso
• Sparse Principal Component Analysis for High Dimensional Vector Autoregressive Models
• The Spectral Norm of Random Inner-Product Kernel Matrices
• Sparse GCA and Thresholded Gradient Descent
• Simpler Proofs for Approximate Factor Models of Large Dimensions
• Learning Feature Sparse Principal Components
• Integrative Factor Regression and Its Inference for Multimodal Data Analysis
• Empirical Bayes PCA in high dimensions
• Euclidean Representation of Low-Rank Matrices and Its Statistical Applications
• Long Random Matrices and Tensor Unfolding
• Sequential scaled sparse factor regression
• On the Worst-Case Approximability of Sparse PCA
• Overlapping community detection in networks via sparse spectral decomposition
• High Dimensional Process Monitoring Using Robust Sparse Probabilistic Principal Component Analysis
• Sparse outlier-robust PCA for multi-source data
• Modularity of the metabolic gene network as a prognostic biomarker for hepatocellular carcinoma
• On estimation of the noise variance in high-dimensional probabilistic principal component analysis
• Optimal Projected Variance Group-Sparse Block PCA
• Nonparametric Trace Regression in High Dimensions via Sign Series Representation
• Large dimensional analysis of general margin based classification methods
• Non-Sparse PCA in High Dimensions via Cone Projected Power Iteration
• Machinery for Proving Sum-of-Squares Lower Bounds on Certification Problems
• One-Bit Compressed Sensing via One-Shot Hard Thresholding
• Bernstein-von Mises Theorems for Functionals of Covariance Matrix
• Convergence rate of eigenvector empirical spectral distribution of large Wigner matrices
• An iterative coordinate descent algorithm to compute sparse low-rank approximations
• Estimating Principal Components under Adversarial Perturbations
• Analysis of Truncated Orthogonal Iteration for Sparse Eigenvector Problems
• Nonparametric principal subspace regression
• A Fast deflation Method for Sparse Principal Component Analysis via Subspace Projections
• Principal component analysis for high-dimensional compositional data
• High Dimensional Semiparametric Scale-Invariant Principal Component Analysis
• On Cross-validation for Sparse Reduced Rank Regression
• The Sparse Principal Component of a Constant-rank Matrix
• Estimation of the number of spikes using a generalized spike population model and application to RNA-seq data
• A Sparse PCA Approach to Clustering