Covariance Estimation: The GLM and Regularization Perspectives

Abstract

Finding an unconstrained and statistically interpretable reparameterization of a covariance matrix is still an open problem in statistics. Its solution is of central importance in covariance estimation, particularly in the recent high-dimensional data environment where enforcing the positive-definiteness constraint could be computationally expensive. We provide a survey of the progress made in modeling covariance matrices from two relatively complementary perspectives: (1) generalized linear models (GLM) or parsimony and use of covariates in low dimensions, and (2) regularization or sparsity for high-dimensional data. An emerging, unifying and powerful trend in both perspectives is that of reducing a covariance estimation problem to that of estimating a sequence of regression problems. We point out several instances of the regression-based formulation. A notable case is in sparse estimation of a precision matrix or a Gaussian graphical model leading to the fast graphical LASSO algorithm. Some advantages and limitations of the regression-based Cholesky decomposition relative to the classical spectral (eigenvalue) and variance-correlation decompositions are highlighted. The former provides an unconstrained and statistically interpretable reparameterization, and guarantees the positive-definiteness of the estimated covariance matrix. It reduces the unintuitive task of covariance estimation to that of modeling a sequence of regressions at the cost of imposing an a priori order among the variables. Elementwise regularization of the sample covariance matrix such as banding, tapering and thresholding has desirable asymptotic properties and the sparse estimated covariance matrix is positive definite with probability tending to one for large samples and dimensions.

Published in at http://dx.doi.org/10.1214/11-STS358 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (20)

• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Rejoinder to "Least angle regression" by Efron et al
• Least Angle Regression
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• Discussion of "Least angle regression" by Efron et al
• High-dimensional graphs and variable selection with the Lasso
• Regularized estimation of large covariance matrices
• Covariance regularization by thresholding
• Sparse permutation invariant covariance estimation
• Sparsistency and rates of convergence in large covariance matrix estimation
• Optimal rates of convergence for covariance matrix estimation
• Network exploration via the adaptive LASSO and SCAD penalties
• Operator norm consistent estimation of large-dimensional sparse covariance matrices
• Sparse estimation of large covariance matrices via a nested Lasso penalty
• Flexible covariance estimation in graphical Gaussian models

Cited by in corpus (18)

• The Gaussian Graphical Model in Cross-sectional and Time-series Data
• Optimal whitening and decorrelation
• Model Selection in Linear Mixed Models
• Ridge Estimation of Inverse Covariance Matrices from High-Dimensional Data
• Multivariate Covariance Generalized Linear Models
• Stable Estimation of a Covariance Matrix Guided by Nuclear Norm Penalties
• Optimizing regularized Cholesky score for order-based learning of Bayesian networks
• On estimating cosmology-dependent covariance matrices
• A comparison of nonlinear extensions to the ensemble Kalman filter: Gaussian Anamorphosis and Two-Step Ensemble Filters
• Introduction to correlation networks: Interdisciplinary approaches beyond thresholding
• Localization in Ensemble Kalman inversion
• Learning a Factor Model via Regularized PCA
• Improving Sequential Latent Variable Models with Autoregressive Flows
• Concentration of a sparse Bayesian model with Horseshoe prior in estimating high-dimensional precision matrix
• Principal Components of Short-term Variability in Venus' UV Albedo
• Nonasymptotic estimation and support recovery for high dimensional sparse covariance matrices
• On Unifying Randomized Methods For Inverse Problems
• Eigengap Sparsity for Covariance Parsimony