Finite sample approximation results for principal component analysis: a matrix perturbation approach

Abstract

Principal component analysis (PCA) is a standard tool for dimensional reduction of a set of $n$ observations (samples), each with $p$ variables. In this paper, using a matrix perturbation approach, we study the nonasymptotic relation between the eigenvalues and eigenvectors of PCA computed on a finite sample of size $n$, and those of the limiting population PCA as $n\to \infty$. As in machine learning, we present a finite sample theorem which holds with high probability for the closeness between the leading eigenvalue and eigenvector of sample PCA and population PCA under a spiked covariance model. In addition, we also consider the relation between finite sample PCA and the asymptotic results in the joint limit $p,n\to \infty$, with $p/n=c$. We present a matrix perturbation view of the "phase transition phenomenon," and a simple linear-algebra based derivation of the eigenvalue and eigenvector overlap in this asymptotic limit. Moreover, our analysis also applies for finite $p,n$ where we show that although there is no sharp phase transition as in the infinite case, either as a function of noise level or as a function of sample size $n$, the eigenvector of sample PCA may exhibit a sharp "loss of tracking," suddenly losing its relation to the (true) eigenvector of the population PCA matrix. This occurs due to a crossover between the eigenvalue due to the signal and the largest eigenvalue due to noise, whose eigenvector points in a random direction.

Published in at http://dx.doi.org/10.1214/08-AOS618 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (2)

• Tracy--Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices
• Sparse Principal Components Analysis

Cited by in corpus (27)

• Principal Component Analysis: A Natural Approach to Data Exploration
• Sparse principal component analysis and iterative thresholding
• Sparse PCA: Optimal rates and adaptive estimation
• Optimal detection of sparse principal components in high dimension
• Asymptotic power of sphericity tests for high-dimensional data
• Minimax sparse principal subspace estimation in high dimensions
• Convergence and prediction of principal component scores in high-dimensional settings
• Optimality and Sub-optimality of PCA I: Spiked Random Matrix Models
• Treelets--An adaptive multi-scale basis for sparse unordered data
• Asymptotic performance of PCA for high-dimensional heteroscedastic data
• On the sample covariance matrix estimator of reduced effective rank population matrices, with applications to fPCA
• Near-Optimal Stochastic Approximation for Online Principal Component Estimation
• Regression on manifolds: Estimation of the exterior derivative
• Fundamental limits of detection in the spiked Wigner model
• Optimal principal component Analysis of STEM XEDS spectrum images
• Do semidefinite relaxations solve sparse PCA up to the information limit?
• On the probability that all eigenvalues of Gaussian, Wishart, and double Wishart random matrices lie within an interval
• Laws of large numbers for eigenvectors and eigenvalues associated to random subspaces in a tensor product
• Biwhitening Reveals the Rank of a Count Matrix
• Estimating the principal components of correlation matrices from all their empirical eigenvectors
• Fast Robust Subspace Tracking via PCA in Sparse Data-Dependent Noise
• When and why are principal component scores a good tool for visualizing high-dimensional data?
• Weak detection in the spiked Wigner model
• On the Distribution of an Arbitrary Subset of the Eigenvalues for some Finite Dimensional Random Matrices
• Estimators for multivariate allometric regression model
• Testing Simultaneous Diagonalizability
• Precise Error Rates for Computationally Efficient Testing