Cleaning large correlation matrices: tools from random matrix theory

Abstract

This review covers recent results concerning the estimation of large covariance matrices using tools from Random Matrix Theory (RMT). We introduce several RMT methods and analytical techniques, such as the Replica formalism and Free Probability, with an emphasis on the Marchenko-Pastur equation that provides information on the resolvent of multiplicatively corrupted noisy matrices. Special care is devoted to the statistics of the eigenvectors of the empirical correlation matrix, which turn out to be crucial for many applications. We show in particular how these results can be used to build consistent "Rotationally Invariant" estimators (RIE) for large correlation matrices when there is no prior on the structure of the underlying process. The last part of this review is dedicated to some real-world applications within financial markets as a case in point. We establish empirically the efficacy of the RIE framework, which is found to be superior in this case to all previously proposed methods. The case of additively (rather than multiplicatively) corrupted noisy matrices is also dealt with in a special Appendix. Several open problems and interesting technical developments are discussed throughout the paper.

165 pages, article submitted to Physics Reports

References in corpus (48)

• Random-Matrix Theory of Quantum Transport
• Many body localization and thermalization in quantum statistical mechanics
• Level-Spacing Distributions and the Airy Kernel
• Noise Dressing of Financial Correlation Matrices
• Quantum many-body systems out of equilibrium
• Shape Fluctuations and Random Matrices
• A Random Matrix Approach to Cross-Correlations in Financial Data
• Random Matrices and Chaos in Nuclear Physics
• Beta ensembles, stochastic Airy spectrum, and a diffusion
• Large Deviations of Extreme Eigenvalues of Random Matrices
• Replica Field Theory for Deterministic Models (II): A Non-Random Spin Glass with Glassy Behavior
• Extreme Value Statistics of Eigenvalues of Gaussian Random Matrices
• Top eigenvalue of a random matrix: large deviations and third order phase transition
• Around the circular law
• Multivariate analysis and Jacobi ensembles: largest eigenvalue, Tracy--Widom limits and rates of convergence
• High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: Risk underestimation
• On the top eigenvalue of heavy-tailed random matrices
• Large Deviations of the Maximum Eigenvalue in Wishart Random Matrices
• Signal and Noise in Correlation Matrix
• Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond
• Noisy Covariance Matrices and Portfolio Optimization II
• Universality of Wigner random matrices: a Survey of Recent Results
• On the Large N Limit of the Itzykson-Zuber Integral
• Rotational invariant estimator for general noisy matrices
• Kullback-Leibler distance as a measure of the information filtered from multivariate data
• Law of addition in random matrix theory
• Universal correlations for deterministic plus random Hamiltonians
• Spectral Moments of Correlated Wishart Matrices
• Level statistics and localization transitions of Lévy matrices
• Non-Stationarity in Financial Time Series and Generic Features
• A simple derivation of the Tracy-Widom distribution of the maximal eigenvalue of a Gaussian unitary random matrix
• Asymptotics of unitary and othogonal matrix integrals
• Independence test for high dimensional data based on regularized canonical correlation coefficients
• Ridge regression and asymptotic minimax estimation over spheres of growing dimension
• Invariant $\beta$-Wishart ensembles, crossover densities and asymptotic corrections to the Marchenko-Pastur law
• Eigenvector dynamics: general theory and some applications
• Efficient Computation of Limit Spectra of Sample Covariance Matrices
• Multiplication law and S transform for non-hermitian random matrices
• Subordination for the sum of two random matrices
• Instanton Approach to Large $N$ Harish-Chandra-Itzykson-Zuber Integrals
• Distribution of the Smallest Eigenvalue in the Correlated Wishart Model
• Eigenvector dynamics under free addition
• A nested factor model for non-linear dependences in stock returns
• Adding and multiplying random matrices: a generalization of Voiculescu's formulae
• Estimation of the Covariance Matrix of Large Dimensional Data
• Estimating the principal components of correlation matrices from all their empirical eigenvectors
• On the large N limit of matrix integrals over the orthogonal group
• A Survey on the Eigenvalues Local Behavior of Large Complex Correlated Wishart Matrices

Cited by in corpus (56)

• Eigenstate Thermalization Hypothesis and Free Probability
• PCA meets RG
• Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data
• Cavity and replica methods for the spectral density of sparse symmetric random matrices
• Low-frequency vibrational spectrum of mean-field disordered systems
• CorrGAN: Sampling Realistic Financial Correlation Matrices Using Generative Adversarial Networks
• Uncovering the Dynamics of Correlation Structures Relative to the Collective Market Motion
• Linear statistics and pushed Coulomb gas at the edge of beta random matrices: four paths to large deviations
• The q-dependent detrended cross-correlation analysis of stock market
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• Introduction to correlation networks: Interdisciplinary approaches beyond thresholding
• On the overlaps between eigenvectors of correlated random matrices
• Scaling in the eigenvalue fluctuations of the empirical correlation matrices
• Matrix Kesten Recursion, Inverse-Wishart Ensemble and Fermions in a Morse Potential
• Gene communities in co-expression networks across different tissues
• Spectra of large time-lagged correlation matrices from Random Matrix Theory
• Semi-metric portfolio optimization: a new algorithm reducing simultaneous asset shocks
• Microcanonical windows on quantum operators
• Random matrix analysis of deep neural network weight matrices
• Covariance matrix filtering with bootstrapped hierarchies
• Gumbel Central Limit Theorem for Max-Min and Min-Max
• Cleaning large-dimensional covariance matrices for correlated samples
• Singular Vectors of Sums of Rectangular Random Matrices and Optimal Estimators of High-Rank Signals: The Extensive Spike Model
• Nonparametric sign prediction of high-dimensional correlation matrix coefficients
• Matrix denoising: Bayes-optimal estimators via low-degree polynomials
• Antagonistic interactions can stabilise fixed points in heterogeneous linear dynamical systems
• A new spin on optimal portfolios and ecological equilibria
• Occupancy correlations in the asymmetric simple inclusion process
• Two-step estimators of high dimensional correlation matrices
• Optimal design of experiments by combining coarse and fine measurements
• Statistical diagonalization of a random biased Hamiltonian: the case of the eigenvectors
• Eigenvalue and Eigenvector Statistics in Time Series Analysis
• Decomposition Pipeline for Large-Scale Portfolio Optimization with Applications to Near-Term Quantum Computing
• Annealed averages in spin and matrix models
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Self-planting: digging holes in rough landscapes
• Random matrix theory and nested clustered portfolios on Mexican markets
• Decorrelation of a leader by the increasing number of followers
• Boundary between noise and information applied to filtering neural network weight matrices
• Random matrix ensemble for the covariance matrix of Ornstein-Uhlenbeck processes with heterogeneous temperatures
• Block-diagonal idiosyncratic covariance estimation in high-dimensional factor models for financial time series
• End-to-End Large Portfolio Optimization for Variance Minimization with Neural Networks through Covariance Cleaning
• A memory-based method to select the number of relevant components in Principal Component Analysis
• Random selection of factors preserves the correlation structure in a linear factor model to a high degree
• Two short pieces around the Wigner problem
• High-dimensional covariance matrix estimators on simulated portfolios with complex structures
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• The Wishart--Rosenzweig--Porter random matrix ensemble
• Appearance of Random Matrix Theory in Deep Learning
• Principal Component Copulas for Capital Modelling and Systemic Risk
• The High-Dimensional Asymptotics of Principal Component Regression
• Inferring financial stock returns correlation from complex network analysis
• Agglomerative Likelihood Clustering
• Community-level Contagion among Diverse Financial Assets
• Generic Features in the Spectral Decomposition of Correlation Matrices
• Gradient descent dynamics and the jamming transition in infinite dimensions