Cleaning large correlation matrices: tools from random matrix theory

Abstract

This review covers recent results concerning the estimation of large covariance matrices using tools from Random Matrix Theory (RMT). We introduce several RMT methods and analytical techniques, such as the Replica formalism and Free Probability, with an emphasis on the Marchenko-Pastur equation that provides information on the resolvent of multiplicatively corrupted noisy matrices. Special care is devoted to the statistics of the eigenvectors of the empirical correlation matrix, which turn out to be crucial for many applications. We show in particular how these results can be used to build consistent "Rotationally Invariant" estimators (RIE) for large correlation matrices when there is no prior on the structure of the underlying process. The last part of this review is dedicated to some real-world applications within financial markets as a case in point. We establish empirically the efficacy of the RIE framework, which is found to be superior in this case to all previously proposed methods. The case of additively (rather than multiplicatively) corrupted noisy matrices is also dealt with in a special Appendix. Several open problems and interesting technical developments are discussed throughout the paper.

165 pages, article submitted to Physics Reports

References in corpus (12)

• Many body localization and thermalization in quantum statistical mechanics
• Large Deviations of Extreme Eigenvalues of Random Matrices
• Extreme Value Statistics of Eigenvalues of Gaussian Random Matrices
• Multivariate analysis and Jacobi ensembles: largest eigenvalue, Tracy--Widom limits and rates of convergence
• High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: Risk underestimation
• Large Deviations of the Maximum Eigenvalue in Wishart Random Matrices
• Kullback-Leibler distance as a measure of the information filtered from multivariate data
• A simple derivation of the Tracy-Widom distribution of the maximal eigenvalue of a Gaussian unitary random matrix
• Independence test for high dimensional data based on regularized canonical correlation coefficients
• Invariant $\beta$-Wishart ensembles, crossover densities and asymptotic corrections to the Marchenko-Pastur law
• Eigenvector dynamics: general theory and some applications
• On the large N limit of matrix integrals over the orthogonal group