Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module

Abstract

We introduce ScannerBit, the statistics and sampling module of the public, open-source global fitting framework GAMBIT. ScannerBit provides a standardised interface to different sampling algorithms, enabling the use and comparison of multiple computational methods for inferring profile likelihoods, Bayesian posteriors, and other statistical quantities. The current version offers random, grid, raster, nested sampling, differential evolution, Markov Chain Monte Carlo (MCMC) and ensemble Monte Carlo samplers. We also announce the release of a new standalone differential evolution sampler, Diver, and describe its design, usage and interface to ScannerBit. We subject Diver and three other samplers (the nested sampler MultiNest, the MCMC GreAT, and the native ScannerBit implementation of the ensemble Monte Carlo algorithm T-Walk) to a battery of statistical tests. For this we use a realistic physical likelihood function, based on the scalar singlet model of dark matter. We examine the performance of each sampler as a function of its adjustable settings, and the dimensionality of the sampling problem. We evaluate performance on four metrics: optimality of the best fit found, completeness in exploring the best-fit region, number of likelihood evaluations, and total runtime. For Bayesian posterior estimation at high resolution, T-Walk provides the most accurate and timely mapping of the full parameter space. For profile likelihood analysis in less than about ten dimensions, we find that Diver and MultiNest score similarly in terms of best fit and speed, outperforming GreAT and T-Walk; in ten or more dimensions, Diver substantially outperforms the other three samplers on all metrics.

arXiv comment51 pages, 25 figures, 2 tables, matches version accepted for publication in EPJC

References in corpus (32)

• FastJet user manual
• Cosmological parameters from CMB and other data: a Monte-Carlo approach
• MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics
• Gauge Singlet Scalars as Cold Dark Matter
• The Minimal Model of Nonbaryonic Dark Matter: A Singlet Scalar
• SOFTSUSY: a program for calculating supersymmetric spectra
• Update on scalar singlet dark matter
• Fast and reliable MCMC for cosmological parameter estimation
• Supersymmetry Without Prejudice
• A Markov Chain Monte Carlo Analysis of the CMSSM
• The impact of priors and observables on parameter inferences in the Constrained MSSM
• Higgs searches and singlet scalar dark matter: Combined constraints from XENON 100 and the LHC
• FlexibleSUSY -- A spectrum generator generator for supersymmetric models
• II: Bayesian Methods for Cosmological Parameter Estimation from Cosmic Microwave Background Measurements
• Multi-Dimensional mSUGRA Likelihood Maps
• Markov Chain Monte Carlo Exploration of Minimal Supergravity with Implications for Dark Matter
• Fittino, a program for determining MSSM parameters from collider observables using an iterative method
• Killing the cMSSM softly
• Combined analysis of effective Higgs portal dark matter models
• A Profile Likelihood Analysis of the Constrained MSSM with Genetic Algorithms
• Direct Constraints on Minimal Supersymmetry from Fermi-LAT Observations of the Dwarf Galaxy Segue 1
• cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation
• Global Study of the Simplest Scalar Phantom Dark Matter Model
• Challenges of Profile Likelihood Evaluation in Multi-Dimensional SUSY Scans
• Predictions for Supersymmetric Particle Masses in the CMSSM using Indirect Experimental and Cosmological Constraints
• The pMSSM10 after LHC Run 1
• Pippi - painless parsing, post-processing and plotting of posterior and likelihood samples
• Fundamental statistical limitations of future dark matter direct detection experiments
• A global fit of the $\gamma$-ray galactic center excess within the scalar singlet Higgs portal model
• A Coverage Study of the CMSSM Based on ATLAS Sensitivity Using Fast Neural Networks Techniques
• Statistical coverage for supersymmetric parameter estimation: a case study with direct detection of dark matter
• Optimization of Neutrino Oscillation Parameters using Differential Evolution

Cited by in corpus (46)

• New physics explanations of $a_{\mu }$ in light of the FNAL muon $g-2$ measurement
• Nested sampling for physical scientists
• Gravitational waves and electroweak baryogenesis in a global study of the extended scalar singlet model
• A Global Analysis of Dark Matter Signals from 27 Dwarf Spheroidal Galaxies using 11 Years of Fermi-LAT Observations
• anesthetic: nested sampling visualisation
• A frequentist analysis of three right-handed neutrinos with GAMBIT
• Structure Formation Models Weaken Limits on WIMP Dark Matter from Dwarf Spheroidal Galaxies
• SpecBit, DecayBit and PrecisionBit: GAMBIT modules for computing mass spectra, particle decay rates and precision observables
• GUT Physics in the era of the LHC
• Cosmological constraints on decaying axion-like particles: a global analysis
• Publishing statistical models: Getting the most out of particle physics experiments
• Strengthening the bound on the mass of the lightest neutrino with terrestrial and cosmological experiments
• Global fits of axion-like particles to XENON1T and astrophysical data
• Constraining New Physics with Borexino Phase-II spectral data
• Fits to Non-Supersymmetric SO(10) Models with Type I and II Seesaw Mechanisms Using Renormalization Group Evolution
• GAMBIT and its Application in the Search for Physics Beyond the Standard Model
• Global fit of pseudo-Nambu-Goldstone Dark Matter
• Resonant or asymmetric: The status of sub-GeV dark matter
• Likelihood analysis of the flavour anomalies and $g-2$ in the general two Higgs doublet model
• A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications
• Axion global fits with Peccei-Quinn symmetry breaking before inflation using GAMBIT
• Nested sampling cross-checks using order statistics
• Running of Fermion Observables in Non-Supersymmetric SO(10) Models
• Simple and statistically sound recommendations for analysing physical theories
• A minimal non-supersymmetric SO(10) model with Peccei--Quinn symmetry
• Two-component scalar dark matter in $Z_{2n}$ scenarios
• The GAMBIT Universal Model Machine: from Lagrangians to Likelihoods
• Korali: Efficient and Scalable Software Framework for Bayesian Uncertainty Quantification and Stochastic Optimization
• Status of Direct Determination of Solar Neutrino Fluxes after Borexino
• Bayesian Model Selection with Future 21cm Observations of The Epoch of Reionisation
• Robustness of dark matter constraints and interplay with collider searches for New Physics
• Global fits of simplified models for dark matter with GAMBIT I. Scalar and fermionic models with s-channel vector mediators
• Active learning BSM parameter spaces
• EasyScan_HEP: a tool for connecting programs to scan the parameter space of physics models
• Exploring phase space with Nested Sampling
• The Muon Magnetic Moment and Physics Beyond the Standard Model
• Global fits of simplified models for dark matter with GAMBIT II. Vector dark matter with an $s$-channel vector mediator
• Phenomenology of extended multiHiggs doublet models with $S_4$ family symmetry
• Thermal WIMPs and the Scale of New Physics: Global Fits of Dirac Dark Matter Effective Field Theories
• Exploring Fine-tuning of the Next-to-Minimal Composite Higgs Model
• Global fits of SUSY at future Higgs factories
• Bayesian WIMP detection with the Cherenkov Telescope Array
• Flavor Symmetries in the Yukawa Sector of Non-Supersymmetric SO(10): Numerical Fits Using Renormalization Group Running
• Exploring the 2HDM with Global Fits in $\text{GAMBIT}$
• A Global Fit of Non-Relativistic Effective Dark Matter Operators Including Solar Neutrinos
• Minimal 4D Composite Higgs Models Under Current LHC Constraints