Bayesian Control Variates for optimal covariance estimation with pairs of simulations and surrogates

Abstract

Predictions of the mean and covariance matrix of summary statistics are critical for confronting cosmological theories with observations, not least for likelihood approximations and parameter inference. The price to pay for accurate estimates is the extreme cost of running $N$-body and hydrodynamics simulations. Approximate solvers, or surrogates, greatly reduce the computational cost but can introduce significant biases, for example in the non-linear regime of cosmic structure growth. We propose "CARPool Bayes", an approach to solve the inference problem for both the means and covariances using a combination of simulations and surrogates. Our framework allows incorporating prior information for the mean and covariance. We derive closed-form solutions for Maximum A Posteriori covariance estimates that are efficient Bayesian shrinkage estimators, guarantee positive semi-definiteness, and can optionally leverage analytical covariance approximations. We discuss choices of the prior and propose a simple procedure for obtaining optimal prior hyperparameter values with a small set of test simulations. We test our method by estimating the covariances of clustering statistics of GADGET-III $N$-body simulations at redshift $z=0.5$ using surrogates from a 100-1000$\times$ faster particle-mesh code. Taking the sample covariance from 15,000 simulations as the truth, and using an empirical Bayes prior with diagonal blocks, our estimator produces nearly identical Fisher matrix contours for $\Lambda$CDM parameters using only $15$ simulations of the non-linear dark matter power spectrum. In this case the number of simulations is so small that the sample covariance would be degenerate. We show cases where even with a naïve prior our method still improves the estimate. Our framework is applicable to a wide range of cosmological and astrophysical problems where fast surrogates are available.

arXiv comment19 pages, 12 Figures

References in corpus (78)

• The cosmological simulation code GADGET-2
• Large-Scale Structure of the Universe and Cosmological Perturbation Theory
• Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
• Large-Scale Galaxy Bias
• Solving Large Scale Structure in Ten Easy Steps with COLA
• The Quijote simulations
• The CAMELS project: Cosmology and Astrophysics with MachinE Learning Simulations
• Massive Lossless Data Compression and Multiple Parameter Estimation from Galaxy Spectra
• The Clustering of Galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Including covariance matrix errors
• HACC: Simulating Sky Surveys on State-of-the-Art Supercomputing Architectures
• AbacusSummit: A Massive Set of High-Accuracy, High-Resolution $N$-Body Simulations
• Learning to Predict the Cosmological Structure Formation
• FastPM: a new scheme for fast simulations of dark matter and halos
• The Effect of Covariance Estimator Error on Cosmological Parameter Constraints
• Putting the Precision in Precision Cosmology: How accurate should your data covariance matrix be?
• The BACCO Simulation Project: Exploiting the full power of large-scale structure for cosmology
• EZmocks: extending the Zel'dovich approximation to generate mock galaxy catalogues with accurate clustering statistics
• Cosmological $N$-body simulations with suppressed variance
• PTHalos: A fast method for generating mock galaxy distributions
• Mock galaxy catalogs using the quick particle mesh method
• Modelling Baryon Acoustic Oscillations with Perturbation Theory and Stochastic Halo Biasing
• COSMOPOWER: emulating cosmological power spectra for accelerated Bayesian inference from next-generation surveys
• High Performance P3M N-body code: CUBEP3M
• The Aemulus Project I: Numerical Simulations for Precision Cosmology
• An accurate tool for the fast generation of dark matter halo catalogs
• The Aemulus Project III: Emulation of the Galaxy Correlation Function
• The Aemulus Project II: Emulating the Halo Mass Function
• Simulations of Baryon Acoustic Oscillations II: Covariance matrix of the matter power spectrum
• GreeM : Massively Parallel TreePM Code for Large Cosmological N-body Simulations
• Dependence of cosmic shear covariances on cosmology - Impact on parameter estimation
• Cosmological constraints from BOSS with analytic covariance matrices
• The Abacus Cosmos: A Suite of Cosmological N-body Simulations
• ICE-COLA: Towards fast and accurate synthetic galaxy catalogues optimizing a quasi $N$-body method
• Generalized massive optimal data compression
• Analytic model for the matter power spectrum, its covariance matrix, and baryonic effects
• Dark Energy Survey Year 3 Results: Covariance Modelling and its Impact on Parameter Estimation and Quality of Fit
• Estimating Cosmological Parameter Covariance
• Comparing approximate methods for mock catalogues and covariance matrices II: Power spectrum multipoles
• Comparing approximate methods for mock catalogues and covariance matrices I: correlation function
• Comparing approximate methods for mock catalogues and covariance matrices III: Bispectrum
• SPECULATOR: Emulating stellar population synthesis for fast and accurate galaxy spectra and photometry
• The Mildly Non-Linear Regime of Structure Formation
• Shrinkage Estimation of the Power Spectrum Covariance Matrix
• The CAMELS Multifield Dataset: Learning the Universe's Fundamental Parameters with Artificial Intelligence
• Fewer Mocks and Less Noise: Reducing the Dimensionality of Cosmological Observables with Subspace Projections
• Super-resolution emulator of cosmological simulations using deep physical models
• Perturbative approach to covariance matrix of the matter power spectrum
• Statistical properties of paired fixed fields
• Cosmic Shear Covariance Matrix in $w$CDM: Cosmology Matters
• Inverted initial conditions: exploring the growth of cosmic structure and voids
• On the insufficiency of arbitrarily precise covariance matrices: non-Gaussian weak lensing likelihoods
• Gravitational Lensing Simulations I : Covariance Matrices and Halo Catalogues
• PINOCCHIO and the hierarchical build-up of dark matter haloes
• Matter Power Spectrum Covariance Matrix from the DEUS-PUR ΛCDM simulations: Mass Resolution and non-Gaussian Errors
• CARPool: fast, accurate computation of large-scale structure statistics by pairing costly and cheap cosmological simulations
• Improving the precision matrix for precision cosmology
• FlowPM: Distributed TensorFlow Implementation of the FastPM Cosmological N-body Solver
• An emulator for the Lyman-$\alpha$ forest in beyond-$\Lambda$CDM cosmologies
• An interpretable machine learning framework for dark matter halo formation
• Learning effective physical laws for generating cosmological hydrodynamics with Lagrangian Deep Learning
• General framework for cosmological dark matter bounds using $N$-body simulations
• RascalC: A Jackknife Approach to Estimating Single and Multi-Tracer Galaxy Covariance Matrices
• Non-linear shrinkage estimation of large-scale structure covariance
• Precision matrix expansion - efficient use of numerical simulations in estimating errors on cosmological parameters
• Estimating the power spectrum covariance matrix with fewer mock samples
• Estimating Covariance Matrices for Two- and Three-Point Correlation Function Moments in Arbitrary Survey Geometries
• Non-Linear Matter Power Spectrum Covariance Matrix Errors and Cosmological Parameter Uncertainties
• Disconnected Covariance of 2-point Functions in Large-Scale Structure
• AMICO galaxy clusters in KiDS-DR3: cosmological constraints from large-scale stacked weak lensing profiles
• CARPool Covariance: Fast, unbiased covariance estimation for large-scale structure observables
• CUBE: An Information-optimized parallel Cosmological $N$-body Algorithm
• The DESI $N$-body Simulation Project -- II. Suppressing sample variance with fast simulations
• A Bayesian method for combining theoretical and simulated covariance matrices for large-scale structure surveys
• Does jackknife scale really matter for accurate large-scale structure covariances?
• Covariance of the redshift-space matter power spectrum after reconstruction
• Reducing the Variance of Redshift Space Distortion Measurements from Mock Galaxy Catalogues with Different Lines of Sight
• Perfectly parallel cosmological simulations using spatial comoving Lagrangian acceleration
• Non-Gaussian error bars in galaxy surveys -- II

Cited by in corpus (4)

• Accurate predictions from small boxes: variance suppression via the Zel'dovich approximation
• Precision Redshift-Space Galaxy Power Spectra using Zel'dovich Control Variates
• Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear dark matter density field
• Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear halo density field