Evaluation of three methods for calculating statistical significance when incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process

Abstract

Hypothesis tests for the presence of new sources of Poisson counts amidst background processes are frequently performed in high energy physics (HEP), gamma ray astronomy (GRA), and other branches of science. While there are conceptual issues already when the mean rate of background is precisely known, the issues are even more difficult when the mean background rate has non-negligible uncertainty. After describing a variety of methods to be found in the HEP and GRA literature, we consider in detail three classes of algorithms and evaluate them over a wide range of parameter space, by the criterion of how close the ensemble-average Type I error rate (rejection of the background-only hypothesis when it is true) compares with the nominal significance level given by the algorithm. We recommend wider use of an algorithm firmly grounded in frequentist tests of the ratio of Poisson means, although for very low counts the over-coverage can be severe due to the effect of discreteness. We extend the studies of Cranmer, who found that a popular Bayesian-frequentist hybrid can undercover severely when taken to high Z values. We also examine the profile likelihood method, which has long been used in GRA and HEP; it provides an excellent approximation in much of the parameter space, as previously studied by Rolke and collaborators.

In v4, added line to Table 1 so that Z_PL is given separately for L_G and L_P; made numerics in Z_PL identical to C++ code which is now posted at http://phystat.org/0811001 . In v3, fixed typos and made other minor changes to v2, so as to be materially the same as version in NIM. v2 was a major revision with new material

References in corpus (10)

• Observation of Top Quark Production in Pbar-P Collisions
• Observation of the Top Quark
• Limits and Confidence Intervals in the Presence of Nuisance Parameters
• An unidentified TeV source in the vicinity of Cygnus OB2
• Observation of TeV Gamma Rays from the Crab Nebula with Milagro Using a New Background Rejection Technique
• Search for High Mass Top Quark Production in p anti-p Collisions at S**(1/2) = 1.8 TeV
• Confidence Intervals and Upper Bounds for Small Signals in the Presence of Background Noise
• On Bayesian Treatment of Systematic Uncertainties in Confidence Interval Calculation
• Frequentist Hypothesis Testing with Background Uncertainty
• Measures of Significance in HEP and Astrophysics

Cited by in corpus (107)

• Asymptotic formulae for likelihood-based tests of new physics
• Search for electroweak production of charginos and sleptons decaying into final states with two leptons and missing transverse momentum in $\sqrt{s}=13$ TeV $pp$ collisions using the ATLAS detector
• Measurement of the top quark-pair production cross section with ATLAS in pp collisions at $\sqrt{s}=7\TeV$
• Searches for electroweak production of supersymmetric particles with compressed mass spectra in $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• A search for a doubly-charged Higgs boson in pp collisions at sqrt(s) = 7 TeV
• Search for squarks and gluinos in final states with jets and missing transverse momentum using 36 fb${}^{-1}$ of $\sqrt{s}$=13 TeV $pp$ collision data with the ATLAS detector
• Measurement of the t-channel single top-quark production cross section in pp collisions at sqrt(s) = 7 TeV with the ATLAS detector
• Vectorlike leptons at the Large Hadron Collider
• Observation of Lambda H-4 hyperhydrogen by decay-pion spectroscopy in electron scattering
• Search for new phenomena in events with at least three photons collected in $pp$ collisions at $\sqrt{s}$ = 8 TeV with the ATLAS detector
• Search for squarks and gluinos in final states with jets and missing transverse momentum using 139 fb${}^{-1}$ of $\sqrt{s}$ =13 TeV $pp$ collision data with the ATLAS detector
• Search for chargino--neutralino pair production in final states with three leptons and missing transverse momentum in $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• Time resolved 2 million year old supernova activity discovered in Earth's microfossil record
• Search for chargino-neutralino production with mass splittings near the electroweak scale in three-lepton final states in $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector
• Search for direct production of electroweakinos in final states with one lepton, missing transverse momentum and a Higgs boson decaying into two $b$-jets in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for charginos and neutralinos in final states with two boosted hadronically decaying bosons and missing transverse momentum in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for top-squark pairs decaying into Higgs or Z bosons in pp collisions at sqrt(s) = 8 TeV
• Prospects for vectorlike leptons at future proton-proton colliders
• Composite Higgs Search at the LHC
• First search for $K^{+}\to {\pi }^{+}\nu \bar{\nu }$ using the decay-in-flight technique
• Sparticles in Motion - getting to the line in compressed scenarios with the Recursive Jigsaw Reconstruction
• Search for long-lived heavy neutral leptons with displaced vertices in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Significance of an excess in a counting experiment: assessing the impact of systematic uncertainties and the case with Gaussian background
• Signatures of vector-like top partners decaying into new neutral scalar or pseudoscalar bosons
• Plotting the Differences Between Data and Expectation
• Search for new phenomena in events with two opposite-charge leptons, jets and missing transverse momentum in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for new phenomena with top quark pairs in final states with one lepton, jets, and missing transverse momentum in $pp$ collisions at $\sqrt{s}$= 13 TeV with the ATLAS detector
• Search for new phenomena in final states with $b$-jets and missing transverse momentum in $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• Production of heavy Higgs bosons and decay into top quarks at the LHC
• Searches for new phenomena in events with two leptons, jets, and missing transverse momentum in 139 fb${}^{-1}$ of $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• Search for direct pair production of sleptons and charginos decaying to two leptons and neutralinos with mass splittings near the $W$-boson mass in $\sqrt{s}=13$TeV $pp$ collisions with the ATLAS detector
• Simplified SIMPs and the LHC
• Search for supersymmetry in events with four or more charged leptons in $139\,\mbox{fb\(^{-1}\)}$ of $\sqrt{s}=13$ TeV $pp$ collisions with the ATLAS detector
• An investigation of the very rare $K^{+}\to {\pi }^{+}\nu \bar{\nu }$ decay
• Probing the Weinberg Operator at Colliders
• Majorana Neutrinos in Same-Sign $W^{\pm }{W}^{\pm }$ Scattering at the LHC: Breaking the TeV Barrier
• Can We Discover Double Higgs Production at the LHC?
• Search for muoproduction of $X(3872)$ at COMPASS and indication of a new state $\tilde{X}(3872)$
• Search for new phenomena in $pp$ collisions in final states with tau leptons, $b$-jets, and missing transverse momentum with the ATLAS detector
• Searches for Dark Matter at the LHC: A Multivariate Analysis in the Mono-$Z$ Channel
• Stau as the Lightest Supersymmetric Particle in R-Parity Violating SUSY Models: Discovery Potential with Early LHC Data
• Search for black holes and other new phenomena in high-multiplicity final states in proton-proton collisions at sqrt(s) = 13 TeV
• Discovery reach for generic supersymmetry at the LHC: MT2 versus missing transverse momentum selections for pMSSM searches
• Search for stealth supersymmetry in events with jets, either photons or leptons, and low missing transverse momentum in pp collisions at 8 TeV
• Criteria for projected discovery and exclusion sensitivities of counting experiments
• Search for top squarks decaying to tau sleptons in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Frequentist Evaluation of Intervals Estimated for a Binomial Parameter and for the Ratio of Poisson Means
• Practical Statistics for the LHC
• Search for a light Higgs boson decaying to two gluons or ss-bar in the radiative decays of Y(1S)
• Signal discovery, limits, and uncertainties with sparse On/Off measurements: an objective Bayesian analysis
• Non-Simplified SUSY: Stau-Coannihilation at LHC and ILC
• Low-mass doubly-charged Higgs bosons at LHC
• Constraints on very high energy gamma-ray emission from the Fermi Bubbles with future ground-based experiments
• Modelling theoretical uncertainties in phenomenological analyses for particle physics
• Search for long-lived particles decaying to a pair of muons in proton-proton collisions at $\sqrt{s}$ = 13 TeV
• Performance and optimization of support vector machines in high-energy physics classification problems
• Solar Axion Search Technique with Correlated Signals from Multiple Detectors
• Search for top squarks in events with a Higgs or $Z$ boson using 139 fb${}^{-1}$ of $pp$ collision data at $\sqrt{s}=13$ TeV with the ATLAS detector
• Search for Nearly Mass-Degenerate Higgsinos Using Low-Momentum Mildly Displaced Tracks in $pp$ Collisions at $\sqrt{s}$ = 13 TeV with the ATLAS Detector
• Search for new physics in events with high jet multiplicity and low missing transverse momentum in proton-proton collisions at sqrt(s) = 8 TeV
• Probing compressed mass spectra in the type-II seesaw model at the LHC
• Search for trilepton resonances from chargino and neutralino pair production in $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector
• Search for a light charged Higgs boson in $t\to H^{\pm }b$ decays, with $H^{\pm }\to cs$, in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Vector-Like Top Quark Production via an Electroweak Dipole Moment at a Muon Collider
• Search for direct production of winos and higgsinos in events with two same-charge leptons or three leptons in $pp$ collision data at $\sqrt{s}=13$ TeV with the ATLAS detector
• Estimating the selection efficiency
• Lectures on Statistics in Theory: Prelude to Statistics in Practice
• Statistical Tools for Imaging Atmospheric Cherenkov Telescopes
• Search for pair production of higgsinos in events with two Higgs bosons and missing transverse momentum in $\sqrt{s}=13$ TeV $pp$ collisions at the ATLAS experiment
• Optimising HEP parameter fits via Monte Carlo weight derivative regression
• Statistical significances and projections for proton decay experiments
• Search for direct production of electroweakinos in final states with one lepton, jets and missing transverse momentum in pp collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Pseudoscalar pair production via off-shell Higgs in composite Higgs models
• Single Production of Vectorlike Y Quarks at the HL-LHC
• Statistics for dark matter subhalo searches in gamma rays from a kinematically constrained population model: Fermi-LAT-like telescopes
• Search for opposite sign muon-tau pair and a b-jet at LHC in the context of flavor anomalies
• Inclusive and differential cross section measurements of $\mathrm{t}\bar{\mathrm{t}}\mathrm{b}\bar{\mathrm{b}}$ production in the lepton+jets channel at $\sqrt{s}$ = 13 TeV
• Searches for CEνNS and Physics beyond the Standard Model using Skipper-CCDs at CONNIE
• Improving Heavy Dijet Resonance Searches Using Jet Substructure at the LHC
• Probing sub-TeV Higgsinos aided by a ML-based top tagger in the context of Trilinear RPV SUSY
• Binomial and ratio-of-Poisson-means frequentist confidence intervals applied to the error evaluation of cut efficiencies
• Projections of Discovery Potentials from Expected Background
• On Bayesian analysis of on-off measurements
• A search for $R$-parity violating supersymmetric decays of the top squark to a $b$-jet and a lepton in $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector
• Search for the reaction $e^{+}{e}^{-}\to {\pi }^{+}{\pi }^{-}{\chi }_{cJ}$ and a charmonium-like structure decaying to ${\chi }_{cJ}{\pi }^{\pm }$ between 4.18 and 4.60 GeV
• Search for cascade decays of charged sleptons and sneutrinos in final states with three leptons and missing transverse momentum in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Probing compressed mass spectra in electroweak supersymmetry with Recursive Jigsaw Reconstruction
• VBS ${\mathrm{W}}^{\pm }{\mathrm{W}}^{\pm }\mathrm{H}$ production at the HL-LHC and a 100 TeV pp-collider
• Studying Same-Sign Top Pair Production in Flavor Changing Scalar Models at the HL-LHC
• Investigating Top-Higgs FCNC Couplings at the FCC-hh
• Prospects of Gluino searches in multi-lepton channels in the light of ongoing LHC RUN-III
• Heavy Higgs Coupled to Vector-like Quarks: Strong CP Problem and Search Prospects at the 14 TeV LHC
• A search for top-squark pair production, in final states containing a top quark, a charm quark and missing transverse momentum, using the 139 fb${}^{-1}$ of $pp$ collision data collected by the ATLAS detector
• Prospect study for measurement of $Hb\bar{b}$ coupling at the LHC and FCC-hh
• Search for supersymmetry in final states with missing transverse momentum and charm-tagged jets using 139 fb${}^{-1}$ of proton-proton collisions at $\sqrt{s}=$ 13 TeV with the ATLAS detector
• A Comparison of Limit Setting Methods for the On-Off Problem
• General search for supersymmetric particles in scenarios with compressed mass spectra using proton-proton collisions at $\sqrt{s}$ = 13 TeV
• Search for long-lived charginos and $\tau$-sleptons using final states with a disappearing track in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• The Faintest, Extremely Variable X-ray Tidal Disruption Event from a Supermassive Black Hole Binary?
• Search for squarks and gluinos in $pp$ collisions at $\sqrt{s}=13$ TeV and $13.6$ TeV in events with $\tau$-leptons, jets and missing transverse momentum using the ATLAS detector
• Vector-Like top quark production via a chromo-magnetic moment at the LHC
• Dark Matter emission at Belle II and NA62 in Minimal Flavor Violation framework
• A Bayesian on-off analysis of cosmic ray data
• Estimation of antihydrogen properties in experiments with small signal deficit
• Significance for signal changes in gamma-ray astronomy
• Alternative signatures of the quintuplet fermions at the LHC and future linear colliders
• A search for small and medium scale anisotropy of cosmic rays above 1 PeV registered at the Tunka-133 array