Simple and statistically sound recommendations for analysing physical theories

Abstract

Physical theories that depend on many parameters or are tested against data from many different experiments pose unique challenges to statistical inference. Many models in particle physics, astrophysics and cosmology fall into one or both of these categories. These issues are often sidestepped with statistically unsound ad hoc methods, involving intersection of parameter intervals estimated by multiple experiments, and random or grid sampling of model parameters. Whilst these methods are easy to apply, they exhibit pathologies even in low-dimensional parameter spaces, and quickly become problematic to use and interpret in higher dimensions. In this article we give clear guidance for going beyond these procedures, suggesting where possible simple methods for performing statistically sound inference, and recommendations of readily-available software tools and standards that can assist in doing so. Our aim is to provide any physicists lacking comprehensive statistical training with recommendations for reaching correct scientific conclusions, with only a modest increase in analysis burden. Our examples can be reproduced with the code publicly available at https://doi.org/10.5281/zenodo.4322283.

arXiv comment15 pages, 4 figures. extended discussions. closely matches version accepted for publication

References in corpus (48)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• emcee: The MCMC Hammer
• Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
• Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC
• Asymptotic formulae for likelihood-based tests of new physics
• A Unified Approach to the Classical Statistical Analysis of Small Signals
• MCMC using Hamiltonian dynamics
• MultiNest: an efficient and robust Bayesian inference tool for cosmology and particle physics
• dynesty: A Dynamic Nested Sampling Package for Estimating Bayesian Posteriors and Evidences
• Bayes in the sky: Bayesian inference and model selection in cosmology
• Planck 2018 results. V. CMB power spectra and likelihoods
• Abandon Statistical Significance
• Limits and Confidence Intervals in the Presence of Nuisance Parameters
• The global electroweak fit at NNLO and prospects for the LHC and ILC
• PolyChord: nested sampling for cosmology
• Trial factors for the look elsewhere effect in high energy physics
• Dark matter direct-detection experiments
• Averages of $b$-hadron, $c$-hadron, and $\tau$-lepton properties as of 2018
• Data analysis recipes: Using Markov Chain Monte Carlo
• A Short History of Markov Chain Monte Carlo: Subjective Recollections from Incomplete Data
• 2000 CKM-Triangle Analysis A Critical Review with Updated Experimental Inputs and Theoretical Parameters
• A Markov Chain Monte Carlo Analysis of the CMSSM
• Constrained MSSM favoring new territories: The impact of new LHC limits and a 125 GeV Higgs boson
• Improved limits on dark matter annihilation in the Sun with the 79-string IceCube detector and implications for supersymmetry
• GAMBIT: The Global and Modular Beyond-the-Standard-Model Inference Tool
• Higgs and Supersymmetry
• Global fits of GUT-scale SUSY models with GAMBIT
• $\text{HEPfit}$: a Code for the Combination of Indirect and Direct Constraints on High Energy Physics Models
• Natural Priors, CMSSM Fits and LHC Weather Forecasts
• Constrained Supersymmetry after two years of LHC data: a global view with Fittino
• ColliderBit: a GAMBIT module for the calculation of high-energy collider observables and likelihoods
• Impact of vacuum stability, perturbativity and XENON1T on global fits of ${\mathbb{Z}}_2$ and ${\mathbb{Z}}_3$ scalar singlet dark matter
• The Jeffreys-Lindley Paradox and Discovery Criteria in High Energy Physics
• Searching for new physics with profile likelihoods: Wilks and beyond
• Comparison of statistical sampling methods with ScannerBit, the GAMBIT scanning module
• Fundamental statistical limitations of future dark matter direct detection experiments
• Use of event-level neutrino telescope data in global fits for theories of new physics
• A Coverage Study of the CMSSM Based on ATLAS Sensitivity Using Fast Neural Networks Techniques
• LikeDM: likelihood calculator of dark matter detection
• Discovering the Significance of 5 sigma
• A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications
• Reproducibility and Replication of Experimental Particle Physics Results
• Statistical coverage for supersymmetric parameter estimation: a case study with direct detection of dark matter
• Frequentist Evaluation of Intervals Estimated for a Binomial Parameter and for the Ratio of Poisson Means
• Comment on "Bayesian Analysis of Pentaquark Signals from CLAS Data", with Response to the Reply by Ireland and Protopopsecu
• HEPLike: an open source framework for experimental likelihood evaluation
• Ordering Algorithms and Confidence Intervals in the Presence of Nuisance Parameters
• Reporting Results in High Energy Physics Publications: a Manifesto

Cited by in corpus (13)

• Cosmological phase transitions: from perturbative particle physics to gravitational waves
• Nested sampling for physical scientists
• Low energy SUSY confronted with new measurements of W-boson mass and muon g-2
• $B$ meson anomalies and large $B^{+}\to K^{+}\nu \bar{\nu }$ in non-universal $U(1{)}^{\prime }$ models
• Hunting WIMPs with LISA: Correlating dark matter and gravitational wave signals
• Active learning BSM parameter spaces
• Exploring phase space with Nested Sampling
• Global fits of SUSY at future Higgs factories
• PEANUTS: a software for the automatic computation of solar neutrino flux and its propagation within Earth
• Relaxing Limits from Big Bang Nucleosynthesis on Heavy Neutral Leptons with Axion-like Particles
• A Global Fit of Non-Relativistic Effective Dark Matter Operators Including Solar Neutrinos
• Explaining the hints for lepton flavour universality violation with three $S_2$ leptoquark generations
• Bring the noise: exact inference from noisy simulations in collider physics