Benchmarking Simulation-Based Inference

Abstract

Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. However, a public benchmark with appropriate performance metrics for such 'likelihood-free' algorithms has been lacking. This has made it difficult to compare algorithms and identify their strengths and weaknesses. We set out to fill this gap: We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms including recent approaches employing neural networks and classical Approximate Bayesian Computation methods. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency. Neural network-based approaches generally exhibit better performance, but there is no uniformly best algorithm. We provide practical advice and highlight the potential of the benchmark to diagnose problems and improve algorithms. The results can be explored interactively on a companion website. All code is open source, making it possible to contribute further benchmark tasks and inference algorithms.

In AISTATS 2021

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems
• Adaptive approximate Bayesian computation
• Non-linear regression models for Approximate Bayesian Computation
• Normalizing Flows for Probabilistic Modeling and Inference
• Data analysis recipes: Using Markov Chain Monte Carlo
• Validating Bayesian Inference Algorithms with Simulation-Based Calibration
• Mining gold from implicit models to improve likelihood-free inference
• Fast $\epsilon$-free Inference of Simulation Models with Bayesian Conditional Density Estimation
• Approximating Likelihood Ratios with Calibrated Discriminative Classifiers
• Automatic physical inference with information maximising neural networks
• A Kernel Test of Goodness of Fit
• cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation
• Learning Deep Kernels for Non-Parametric Two-Sample Tests
• Automatic Posterior Transformation for Likelihood-Free Inference
• ELFI: Engine for Likelihood-Free Inference
• Inference Compilation and Universal Probabilistic Programming
• On Contrastive Learning for Likelihood-free Inference
• Dynamic Likelihood-free Inference via Ratio Estimation (DIRE)
• Approximating the Likelihood in Approximate Bayesian Computation
• Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations

Cited by in corpus (5)

• Bayesian Stokes inversion with Normalizing flows
• Neural Approximate Sufficient Statistics for Implicit Models
• Sequential Neural Posterior and Likelihood Approximation
• Optimal Design of Experiments for Simulation-Based Inference of Mechanistic Acyclic Biological Networks
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories