A Trust Crisis In Simulation-Based Inference? Your Posterior Approximations Can Be Unfaithful

Abstract

We present extensive empirical evidence showing that current Bayesian simulation-based inference algorithms can produce computationally unfaithful posterior approximations. Our results show that all benchmarked algorithms -- (Sequential) Neural Posterior Estimation, (Sequential) Neural Ratio Estimation, Sequential Neural Likelihood and variants of Approximate Bayesian Computation -- can yield overconfident posterior approximations, which makes them unreliable for scientific use cases and falsificationist inquiry. Failing to address this issue may reduce the range of applicability of simulation-based inference. For this reason, we argue that research efforts should be made towards theoretical and methodological developments of conservative approximate inference algorithms and present research directions towards this objective. In this regard, we show empirical evidence that ensembling posterior surrogates provides more reliable approximations and mitigates the issue.

TMLR version

Cited by in corpus (14)

• EFTofLSS meets simulation-based inference: ${\sigma }_8$ from biased tracers
• One never walks alone: the effect of the perturber population on subhalo measurements in strong gravitational lenses
• Scalable inference with Autoregressive Neural Ratio Estimation
• Neural Simulation-Based Inference of the Neutron Star Equation of State directly from Telescope Spectra
• Fast Likelihood-free Reconstruction of Gravitational Wave Backgrounds
• Simulation-based inference of single-molecule experiments
• Fast likelihood-free inference in the LSS Stage IV era
• Testing the robustness of simulation-based gravitational-wave population inference
• Simulation-based Inference for Gravitational-waves from Intermediate-Mass Binary Black Holes in Real Noise
• Simulation-based Inference of Developmental EEG Maturation with the Spectral Graph Model
• Physically Interpretable Probabilistic Domain Characterization
• Deep inference of simulated strong lenses in ground-based surveys
• Flexible Simulation Based Inference for Galaxy Photometric Fitting with Synthesizer
• FlowSN: Neural Simulation-Based Inference under Realistic Selection Effects applied to Supernova Cosmology