High Dimensional Statistical Inference and Random Matrices

Abstract

Multivariate statistical analysis is concerned with observations on several variables which are thought to possess some degree of inter-dependence. Driven by problems in genetics and the social sciences, it first flowered in the earlier half of the last century. Subsequently, random matrix theory (RMT) developed, initially within physics, and more recently widely in mathematics. While some of the central objects of study in RMT are identical to those of multivariate statistics, statistical theory was slow to exploit the connection. However, with vast data collection ever more common, data sets now often have as many or more variables than the number of individuals observed. In such contexts, the techniques and results of RMT have much to offer multivariate statistics. The paper reviews some of the progress to date.

27 pages, 5 figures, to appear in Proc. International Congress of Mathematicians, 2006

References in corpus (11)

• Level-Spacing Distributions and the Airy Kernel
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Statistical Challenges with High Dimensionality: Feature Selection in Knowledge Discovery
• Beta ensembles, stochastic Airy spectrum, and a diffusion
• Sparse Principal Components Analysis
• Classical skew orthogonal polynomials and random matrices
• On the top eigenvalue of heavy-tailed random matrices
• A rate of convergence result for the largest eigenvalue of complex white Wishart matrices
• On the largest eigenvalue of Wishart matrices with identity covariance when n, p and p/n tend to infinity
• Universality for mathematical and physical systems
• Numerical Methods for Eigenvalue Distributions of Random Matrices

Cited by in corpus (49)

• OptShrink: An algorithm for improved low-rank signal matrix denoising by optimal, data-driven singular value shrinkage
• Universality of covariance matrices
• Distribution of the largest eigenvalue for real Wishart and Gaussian random matrices and a simple approximation for the Tracy-Widom distribution
• Limits of spiked random matrices I
• Universality for the largest eigenvalue of sample covariance matrices with general population
• Edge universality of correlation matrices
• A necessary and sufficient condition for edge universality at the largest singular values of covariance matrices
• Testing for Independence of Large Dimensional Vectors
• Efficient Computation of Limit Spectra of Sample Covariance Matrices
• Sharp detection in PCA under correlations: all eigenvalues matter
• Cross correlations of the American baby names
• Single-lead f-wave extraction using diffusion geometry
• Edge universality of separable covariance matrices
• Geometric sensitivity of random matrix results: consequences for shrinkage estimators of covariance and related statistical methods
• Convergence of eigenvector empirical spectral distribution of sample covariance matrices
• Exact and Stable Covariance Estimation from Quadratic Sampling via Convex Programming
• Asymptotic behavior of large Gaussian correlated Wishart matrices
• The bootstrap, covariance matrices and PCA in moderate and high-dimensions
• Denoising Click-evoked Otoacoustic Emission Signals by Optimal Shrinkage
• Operator limits of random matrices
• The Correlated Jacobi and the Correlated Cauchy-Lorentz ensembles
• Regularization in High-Dimensional Regression and Classification via Random Matrix Theory
• Threshold Effects in Parameter Estimation from Compressed Data
• Linear spectral statistics of eigenvectors of anisotropic sample covariance matrices
• Bayesian inference for spectral projectors of the covariance matrix
• Fluctuations of eigenvalues and second order Poincaré inequalities
• When are the most informative components for inference also the principal components?
• CLT for linear spectral statistics of large dimensional sample covariance matrices with dependent data
• Marchenko-Pastur Law for Tyler's M-estimator
• Detection of Signal in the Spiked Rectangular Models
• CLT for large dimensional general Fisher matrices and its applications in high-dimensional data analysis
• CLT for linear spectral statistics of random matrix $S^{-1}T$
• Substitution principle for CLT of linear spectral statistics of high-dimensional sample covariance matrices with applications to hypothesis testing
• Large deviations of spread measures for Gaussian matrices
• Think globally, fit locally under the Manifold Setup: Asymptotic Analysis of Locally Linear Embedding
• Central Limit Theorem for Linear Spectral Statistics of Large Dimensional Kendall's Rank Correlation Matrices and its Applications
• Asymptotic Coincidence of the Statistics for Degenerate and Non-Degenerate Correlated Real Wishart Ensembles
• Fundamental limit of sample generalized eigenvalue based detection of signals in noise using relatively few signal-bearing and noise-only samples
• On eigenvalues of a high-dimensional spatial-sign covariance matrix
• Central Limit Theorem for Linear Eigenvalue Statistics for Submatrices of Wigner Random Matrices
• Finite sample Bernstein-von Mises theorems for functionals and spectral projectors of the covariance matrix
• Statistical and knowledge supported visualization of multivariate data
• Inference of principal components of noisy correlation matrices with prior information
• Linear spectral statistics of sequential sample covariance matrices
• Nonparametric Learning of Two-Layer ReLU Residual Units
• Transient-evoked otoacoustic emission signals predicting outcomes of acute sensorineural hearing loss in patients with Meniere's Disease
• Optimal Recovery of Precision Matrix for Mahalanobis Distance from High Dimensional Noisy Observations in Manifold Learning
• Sample eigenvalue based detection of high dimensional signals in white noise using relatively few samples
• Eigenvalues of sample covariance matrices of non-linear processes with infinite variance