Massive data compression for parameter-dependent covariance matrices

Abstract

We show how the massive data compression algorithm MOPED can be used to reduce, by orders of magnitude, the number of simulated datasets that are required to estimate the covariance matrix required for the analysis of gaussian-distributed data. This is relevant when the covariance matrix cannot be calculated directly. The compression is especially valuable when the covariance matrix varies with the model parameters. In this case, it may be prohibitively expensive to run enough simulations to estimate the full covariance matrix throughout the parameter space. This compression may be particularly valuable for the next-generation of weak lensing surveys, such as proposed for Euclid and LSST, for which the number of summary data (such as band power or shear correlation estimates) is very large, $\sim 10^4$, due to the large number of tomographic redshift bins that the data will be divided into. In the pessimistic case where the covariance matrix is estimated separately for all points in an MCMC analysis, this may require an unfeasible $10^9$ simulations. We show here that MOPED can reduce this number by a factor of 1000, or a factor of $\sim 10^6$ if some regularity in the covariance matrix is assumed, reducing the number of simulations required to a manageable $10^3$, making an otherwise intractable analysis feasible.

8 pages. Accepted by MNRAS

References in corpus (28)

• Karhunen-Loeve eigenvalue problems in cosmology: how should we tackle large data sets?
• Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
• CFHTLenS tomographic weak lensing cosmological parameter constraints: Mitigating the impact of intrinsic galaxy alignments
• The Complete Star Formation History of the Universe
• The Coyote Universe I: Precision Determination of the Nonlinear Matter Power Spectrum
• The star formation histories of galaxies in the Sloan Digital Sky Survey
• Massive Lossless Data Compression and Multiple Parameter Estimation from Galaxy Spectra
• The Clustering of Galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Including covariance matrix errors
• Parameter inference with estimated covariance matrices
• The Coyote Universe II: Cosmological Models and Precision Emulation of the Nonlinear Matter Power Spectrum
• The Effect of Covariance Estimator Error on Cosmological Parameter Constraints
• Estimating Cosmological Parameter Covariance
• Recovering Physical Parameters from Galaxy Spectra Using MOPED
• Shrinkage Estimation of the Power Spectrum Covariance Matrix
• Sample variance in weak lensing: how many simulations are required?
• Quantifying lost information due to covariance matrix estimation in parameter inference
• Non-linear shrinkage estimation of large-scale structure covariance
• Precision matrix expansion - efficient use of numerical simulations in estimating errors on cosmological parameters
• On estimating cosmology-dependent covariance matrices
• Non-Linear Matter Power Spectrum Covariance Matrix Errors and Cosmological Parameter Uncertainties
• Combining Size and Shape in Weak Lensing
• Estimating sparse precision matrices
• Linear acceleration emission in pulsar magnetospheres
• Extreme data compression for the CMB
• A New Data Compression Method and its Application to Cosmic Shear Analysis
• Variations of cosmic large-scale structure covariance matrices across parameter space
• Fast Parameter Estimation from the CMB Power Spectrum
• An investigation into the Multiple Optimised Parameter Estimation and Data compression algorithm

Cited by in corpus (47)

• Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
• Automatic physical inference with information maximising neural networks
• Generalized massive optimal data compression
• Bayesian optimisation for likelihood-free cosmological inference
• Scientific Synergy Between LSST and Euclid
• Fewer Mocks and Less Noise: Reducing the Dimensionality of Cosmological Observables with Subspace Projections
• Dark Energy Survey Year 3 results: cosmology with moments of weak lensing mass maps -- validation on simulations
• KiDS+GAMA: Constraints on Horndeski gravity from combined large-scale structure probes
• Maximal compression of the redshift space galaxy power spectrum and bispectrum
• Enhancing BOSS bispectrum cosmological constraints with maximal compression
• The skewed weak lensing likelihood: why biases arise, despite data and theory being sound
• Lossless, Scalable Implicit Likelihood Inference for Cosmological Fields
• Robust Field-level Likelihood-free Inference with Galaxies
• Data compression in cosmology: A compressed likelihood for Planck data
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• Persistent homology in cosmic shear II: A tomographic analysis of DES-Y1
• Capse.jl: efficient and auto-differentiable CMB power spectra emulation
• Cosmology from weak lensing peaks and minima with Subaru Hyper Suprime-Cam survey first-year data
• Beyond two-point statistics: using the minimum spanning tree as a tool for cosmology
• Robust Simulation-Based Inference in Cosmology with Bayesian Neural Networks
• Euclid-era cosmology for everyone: Neural net assisted MCMC sampling for the joint 3x2 likelihood
• Covariance of the galaxy angular power spectrum with the halo model
• New constraints on cosmological modified gravity theories from anisotropic three-point correlation functions of BOSS DR12 galaxies
• Non-Gaussianity in the Weak Lensing Correlation Function Likelihood -- Implications for Cosmological Parameter Biases
• Parameter Inference for Weak Lensing using Gaussian Processes and MOPED
• candl: Cosmic Microwave Background Analysis with a Differentiable Likelihood
• A Bayesian method for combining theoretical and simulated covariance matrices for large-scale structure surveys
• Constraints on features in the inflationary potential from future Euclid data
• Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
• Extreme data compression while searching for new physics
• Indra: a Public Computationally Accessible Suite of Cosmological $N$-body Simulations
• Cosmological constraints using Minkowski functionals from the first year data of the Hyper Suprime-Cam
• KiDS-Legacy: Covariance validation and the unified OneCovariance framework for projected large-scale structure observables
• Field-level simulation-based inference with galaxy catalogs: the impact of systematic effects
• The impact of signal-to-noise, redshift, and angular range on the bias of weak lensing 2-point functions
• Dark Energy Survey Year 3 Results: Cosmological constraints from second and third-order shear statistics
• Science-driven 3D data compression
• Faster cosmological analysis with power spectrum without simulations
• Cosmological inference including massive neutrinos from the matter power spectrum: biases induced by uncertainties in the covariance matrix
• Data Compression and Covariance Matrix Inspection: Cosmic Shear
• Identifying the most constraining ice observations to infer molecular binding energies
• Speeding up the detectability of the harmonic-space galaxy bispectrum
• Skew-spectra: a generalization to spin-$s$
• $\text{SBi3PCF:}$ Simulation-based inference with the integrated 3PCF
• The Parameter-Level Performance of Covariance Matrix Conditioning in Cosmic Microwave Background Data Analyses
• Origin & evolution of the Universe
• Improving cosmological covariance matrices with machine learning