Validating Bayesian Inference Algorithms with Simulation-Based Calibration

Abstract

Verifying the correctness of Bayesian computation is challenging. This is especially true for complex models that are common in practice, as these require sophisticated model implementations and algorithms. In this paper we introduce \emph{simulation-based calibration} (SBC), a general procedure for validating inferences from Bayesian algorithms capable of generating posterior samples. This procedure not only identifies inaccurate computation and inconsistencies in model implementations but also provides graphical summaries that can indicate the nature of the problems that arise. We argue that SBC is a critical part of a robust Bayesian workflow, as well as being a useful tool for those developing computational algorithms and statistical software.

arXiv comment19 pages, 13 figures

References in corpus (3)

• The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
• A Conceptual Introduction to Hamiltonian Monte Carlo
• A General Framework for the Parametrization of Hierarchical Models

Cited by in corpus (63)

• Tests of General Relativity with Binary Black Holes from the second LIGO-Virgo Gravitational-Wave Transient Catalog
• Bayesian inference for compact binary coalescences with BILBY: Validation and application to the first LIGO--Virgo gravitational-wave transient catalogue
• Collaborative Nested Sampling: Big Data vs. complex physical models
• Nested sampling for physical scientists
• Nested Sampling with Normalising Flows for Gravitational-Wave Inference
• Efficient Bayesian Structural Equation Modeling in Stan
• Bilby-MCMC: An MCMC sampler for gravitational-wave inference
• Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows
• Rapid Parameter Estimation of Gravitational Waves from Binary Neutron Star Coalescence using Focused Reduced Order Quadrature
• Black hole spectroscopy horizons for current and future gravitational wave detectors
• Simulation-Based Calibration Checking for Bayesian Computation: The Choice of Test Quantities Shapes Sensitivity
• Exploring the likelihood of the 21-cm power spectrum with simulation-based inference
• OutbreakFlow: Model-based Bayesian inference of disease outbreak dynamics with invertible neural networks and its application to the COVID-19 pandemics in Germany
• LtU-ILI: An All-in-One Framework for Implicit Inference in Astrophysics and Cosmology
• Black-box Bayesian inference for economic agent-based models
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality
• Towards constraining warm dark matter with stellar streams through neural simulation-based inference
• Likelihood-free MCMC with Amortized Approximate Ratio Estimators
• Analyzing black-hole ringdowns
• EFTofLSS meets simulation-based inference: ${\sigma }_8$ from biased tracers
• Not All Requirements Prioritization Criteria Are Equal at All Times: A Quantitative Analysis
• Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior
• Gaussian Processes for Glitch-robust Gravitational-wave Astronomy
• Benchmarking Simulation-Based Inference
• Bayesian Analysis of a Future Beta Decay Experiment's Sensitivity to Neutrino Mass Scale and Ordering
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• DIGS: Deep Inference of Galaxy Spectra with Neural Posterior Estimation
• Combining probability distributions: Extending the logarithmic pooling approach
• The effect of spin mismodeling on gravitational-wave measurements of the binary neutron star mass distribution
• Optimal, fast, and robust inference of reionization-era cosmology with the 21cmPIE-INN
• Opaque prior distributions in Bayesian latent variable models
• Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond
• Neural Density Estimation and Likelihood-free Inference
• Diagnostics for Conditional Density Models and Bayesian Inference Algorithms
• AMOEBA: Automated Molecular Excitation Bayesian Line-Fitting Algorithm
• Modelling variability power spectra of active galaxies from irregular time series
• Sequential Neural Posterior and Likelihood Approximation
• Calibration procedures for approximate Bayesian credible sets
• A Simulation Based Inference Approach to the Dynamics of the MW-LMC System -- Validation
• Testing whether a Learning Procedure is Calibrated
• Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference Setting
• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
• Probabilistic Iterative Methods for Linear Systems
• A Bayesian brain model of adaptive behavior: An application to the Wisconsin Card Sorting Task
• A Log-Gaussian Cox Process with Sequential Monte Carlo for Line Narrowing in Spectroscopy
• Amortized Bayesian Inference for Models of Cognition
• Bayesian inference under model misspecification using transport-Lagrangian distances: an application to seismic inversion
• Assessment and adjustment of approximate inference algorithms using the law of total variance
• The experiment is just as important as the likelihood in understanding the prior: A cautionary note on robust cognitive modelling
• Stratified sampling and bootstrapping for approximate Bayesian computation
• Distortion estimates for approximate Bayesian inference
• Static Analysis for Probabilistic Programs
• Quantifying the Milky Way, LMC and their interaction using all-sky kinematics of outer halo stars
• Flexible Simulation Based Inference for Galaxy Photometric Fitting with Synthesizer
• Deep inference of simulated strong lenses in ground-based surveys
• Statistical Estimation of Mean Lorentzian Line Width in Spectra by Gaussian Processes
• Gaussian distributional structural equation models: A framework for modeling latent heteroscedasticity
• Bayesian inference for network Poisson models
• Unit Testing for MCMC and other Monte Carlo Methods
• An Evaluation Framework for Personalization Strategy Experiment Designs
• GILBO: One Metric to Measure Them All
• Simulation based inference of the ionization history from the 2D 21 cm power spectrum
• Validating Gaussian Process Models with Simulation-Based Calibration