Non-linear shrinkage estimation of large-scale structure covariance

Abstract

In many astrophysical settings covariance matrices of large datasets have to be determined empirically from a finite number of mock realisations. The resulting noise degrades inference and precludes it completely if there are fewer realisations than data points. This work applies a recently proposed non-linear shrinkage estimator of covariance to a realistic example from large-scale structure cosmology. After optimising its performance for the usage in likelihood expressions, the shrinkage estimator yields subdominant bias and variance comparable to that of the standard estimator with a factor $\sim 50$ less realisations. This is achieved without any prior information on the properties of the data or the structure of the covariance matrix, at negligible computational cost.

arXiv comment5 pages, 5 figures; accepted for publication in MNRAS

References in corpus (24)

• Karhunen-Loeve eigenvalue problems in cosmology: how should we tackle large data sets?
• Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
• Cosmology with cosmic shear observations: a review
• Nonlinear shrinkage estimation of large-dimensional covariance matrices
• Statistical Analysis of Galaxy Surveys - I. Robust error estimation for 2-point clustering statistics
• Cosmological parameter constraints from galaxy-galaxy lensing and galaxy clustering with the SDSS DR7
• The Clustering of Galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Including covariance matrix errors
• Parameter inference with estimated covariance matrices
• Putting the Precision in Precision Cosmology: How accurate should your data covariance matrix be?
• The Effect of Covariance Estimator Error on Cosmological Parameter Constraints
• Improving lognormal models for cosmological fields
• Cosmic shear covariance: The log-normal approximation
• Estimating Cosmological Parameter Covariance
• Shrinkage Estimation of the Power Spectrum Covariance Matrix
• Performance of internal Covariance Estimators for Cosmic Shear Correlation Functions
• Galaxy and Mass Assembly (GAMA): Redshift Space Distortions from the Clipped Galaxy Field
• Improving the precision matrix for precision cosmology
• Sample variance in weak lensing: how many simulations are required?
• Large Covariance Matrices: Smooth Models from the 2-Point Correlation Function
• Fast generation of ensembles of cosmological N-body simulations via mode-resampling
• Estimating the power spectrum covariance matrix with fewer mock samples
• Estimating sparse precision matrices
• Extreme data compression for the CMB
• Jackknife resampling technique on mocks: an alternative method for covariance matrix estimation

Cited by in corpus (30)

• Galaxy Power Spectrum Multipoles Covariance in Perturbation Theory
• Cosmological constraints from BOSS with analytic covariance matrices
• Dark Energy Survey Year 3 Results: Covariance Modelling and its Impact on Parameter Estimation and Quality of Fit
• 2D-FFTLog: Efficient computation of real space covariance matrices for galaxy clustering and weak lensing
• Dark matter statistics for large galaxy catalogs: power spectra and covariance matrices
• Characterizing Signal Loss in the 21 cm Reionization Power Spectrum: A Revised Study of PAPER-64
• Fewer Mocks and Less Noise: Reducing the Dimensionality of Cosmological Observables with Subspace Projections
• Massive data compression for parameter-dependent covariance matrices
• ENCORE: An $\mathcal{O}(N_{\mathrm{g}}^2)$ Estimator for Galaxy $N$-Point Correlation Functions
• The integrated 3-point correlation function of cosmic shear
• RascalC: A Jackknife Approach to Estimating Single and Multi-Tracer Galaxy Covariance Matrices
• Precision matrix expansion - efficient use of numerical simulations in estimating errors on cosmological parameters
• Large Covariance Matrices: Accurate Models Without Mocks
• Cosmological Information in Skew Spectra of Biased Tracers in Redshift Space
• Non-Gaussianity in the Weak Lensing Correlation Function Likelihood -- Implications for Cosmological Parameter Biases
• A First Detection of the Connected 4-Point Correlation Function of Galaxies Using the BOSS CMASS Sample
• A Bayesian method for combining theoretical and simulated covariance matrices for large-scale structure surveys
• Modal compression of the redshift-space galaxy bispectrum
• Accurate halo-galaxy mocks from automatic bias estimation and particle mesh gravity solvers
• Towards cosmological constraints from the compressed modal bispectrum: a robust comparison of real-space bispectrum estimators
• Fitting covariance matrix models to simulations
• Bayesian Control Variates for optimal covariance estimation with pairs of simulations and surrogates
• Cosmological Model Parameter Dependence of the Matter Power Spectrum Covariance from the DEUS-PUR $Cosmo$ Simulations
• KiDS-Legacy: Covariance validation and the unified OneCovariance framework for projected large-scale structure observables
• Cosmological inference including massive neutrinos from the matter power spectrum: biases induced by uncertainties in the covariance matrix
• A Test for Cosmological Parity Violation Using the 3D Distribution of Galaxies
• Euclid: Fast two-point correlation function covariance through linear construction
• A comparison of shrinkage estimators of the cosmological precision matrix
• Super sample covariance and the volume scaling of galaxy survey covariance matrices
• Residual Smoothing: Using Mocks to Correct Model Covariance Matrices