Simulation-based inference has its own Dodelson-Schneider effect (but it knows that it does)

Abstract

Making inferences about physical properties of the Universe requires knowledge of the data likelihood. A Gaussian distribution is commonly assumed for the uncertainties with a covariance matrix estimated from a set of simulations. The noise in such covariance estimates causes two problems: it distorts the width of the parameter contours, and it adds scatter to the location of those contours which is not captured by the widths themselves. For non-Gaussian likelihoods, an approximation may be derived via Simulation-Based Inference (SBI). It is often implicitly assumed that parameter constraints from SBI analyses, which do not use covariance matrices, are not affected by the same problems as parameter estimation with a covariance matrix estimated from simulations. We investigate whether SBI suffers from effects similar to those of covariance estimation in Gaussian likelihoods. We use Neural Posterior and Likelihood Estimation with continuous and masked autoregressive normalizing flows for density estimation. We fit our approximate posterior models to simulations drawn from a Gaussian linear model, so that the SBI result can be compared to the true posterior. We test linear and neural network based compression, demonstrating that neither methods circumvent the issues of covariance estimation. SBI suffers an inflation of posterior variance that is equal or greater than the analytical result in covariance estimation for Gaussian likelihoods for the same number of simulations. The assumption that SBI requires a smaller number of simulations than covariance estimation for a Gaussian likelihood analysis is inaccurate. The limitations of traditional likelihood analysis with simulation-based covariance remain for SBI with a finite simulation budget. Despite these issues, we show that SBI correctly draws the true posterior contour given enough simulations.

21 pages, 10 figures

References in corpus (55)

• LSST: from Science Drivers to Reference Design and Anticipated Data Products
• The Dark Energy Survey: more than dark energy - an overview
• Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
• Cosmology with cosmic shear observations: a review
• Dark Energy Survey Year 3 Results: Cosmology from Cosmic Shear and Robustness to Data Calibration
• Statistical Analysis of Galaxy Surveys - I. Robust error estimation for 2-point clustering statistics
• The Quijote simulations
• The CAMELS project: Cosmology and Astrophysics with MachinE Learning Simulations
• Massive Lossless Data Compression and Multiple Parameter Estimation from Galaxy Spectra
• CosmoLike - Cosmological Likelihood Analyses for Photometric Galaxy Surveys
• The Clustering of Galaxies in the SDSS-III Baryon Oscillation Spectroscopic Survey: Including covariance matrix errors
• Parameter inference with estimated covariance matrices
• Analysis of two-point statistics of cosmic shear: I. Estimators and covariances
• The Effect of Covariance Estimator Error on Cosmological Parameter Constraints
• Putting the Precision in Precision Cosmology: How accurate should your data covariance matrix be?
• Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
• Cosmology constraints from shear peak statistics in Dark Energy Survey Science Verification data
• Weak Gravitational Lensing
• KiDS-1000 Methodology: Modelling and inference for joint weak gravitational lensing and spectroscopic galaxy clustering analysis
• Likelihood-free inference with neural compression of DES SV weak lensing map statistics
• Density split statistics: Cosmological constraints from counts and lensing in cells in DES Y1 and SDSS data
• Nuisance hardened data compression for fast likelihood-free inference
• Towards an Optimal Estimation of Cosmological Parameters with the Wavelet Scattering Transform
• Cosmology with the Wide-Field Infrared Survey Telescope -- Multi-Probe Strategies
• Fisher for complements: Extracting cosmology and neutrino mass from the counts-in-cells PDF
• Automatic physical inference with information maximising neural networks
• Likelihood-Free Cosmological Inference with Type Ia Supernovae: Approximate Bayesian Computation for a Complete Treatment of Uncertainty
• Generalized massive optimal data compression
• Dark Energy Survey Year 3 Results: Covariance Modelling and its Impact on Parameter Estimation and Quality of Fit
• Density split statistics: joint model of counts and lensing in cells
• Approximate Bayesian Computation for Forward Modeling in Cosmology
• Estimating Cosmological Parameter Covariance
• Precision cosmology with voids in the final BOSS data
• Bayesian optimisation for likelihood-free cosmological inference
• A new model to predict weak-lensing peak counts II. Parameter constraint strategies
• Performance of internal Covariance Estimators for Cosmic Shear Correlation Functions
• Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation
• Cosmological constraints from the BOSS DR12 void size function
• The integrated 3-point correlation function of cosmic shear
• Precision matrix expansion - efficient use of numerical simulations in estimating errors on cosmological parameters
• Exploring the likelihood of the 21-cm power spectrum with simulation-based inference
• Creating Jackknife and Bootstrap estimates of the covariance matrix for the two-point correlation function
• $\mathrm{S}\mathrm{IM}\mathrm{BIG}$: Mock Challenge for a Forward Modeling Approach to Galaxy Clustering
• On the accuracy and precision of correlation functions and field-level inference in cosmology
• EFTofLSS meets simulation-based inference: ${\sigma }_8$ from biased tracers
• A Bayesian method for combining theoretical and simulated covariance matrices for large-scale structure surveys
• How informative are summaries of the cosmic 21-cm signal?
• A simulation-based inference pipeline for cosmic shear with the Kilo-Degree Survey
• Including parameter dependence in the data and covariance for cosmological inference
• Extreme data compression while searching for new physics
• Cosmology from weak lensing, galaxy clustering, CMB lensing and tSZ: I. 10x2pt Modelling Methodology
• Cosmological forecasts with the clustering of weak lensing peaks
• Cosmological Parameter Estimation and Inference using Deep Summaries
• Likelihood-free Forward Modeling for Cluster Weak Lensing and Cosmology
• A roadmap to cosmological parameter analysis with third-order shear statistics II: Analytic covariance estimate

Cited by in corpus (1)

• $\text{SBi3PCF:}$ Simulation-based inference with the integrated 3PCF