Significance of an excess in a counting experiment: assessing the impact of systematic uncertainties and the case with Gaussian background

Abstract

Several experiments in high-energy physics and astrophysics can be treated as on/off measurements, where an observation potentially containing a new source or effect ("on" measurement) is contrasted with a background-only observation free of the effect ("off" measurement). In counting experiments, the significance of the new source or effect can be estimated with a widely-used formula from [LiMa], which assumes that both measurements are Poisson random variables. In this paper we study three other cases: i) the ideal case where the background measurement has no uncertainty, which can be used to study the maximum sensitivity that an instrument can achieve, ii) the case where the background estimate $b$ in the off measurement has an additional systematic uncertainty, and iii) the case where $b$ is a Gaussian random variable instead of a Poisson random variable. The latter case applies when $b$ comes from a model fitted on archival or ancillary data, or from the interpolation of a function fitted on data surrounding the candidate new source/effect. Practitioners typically use in this case a formula which is only valid when $b$ is large and when its uncertainty is very small, while we derive a general formula that can be applied in all regimes. We also develop simple methods that can be used to assess how much an estimate of significance is sensitive to systematic uncertainties on the efficiency or on the background. Examples of applications include the detection of short Gamma-Ray Bursts and of new X-ray or $\gamma$-ray sources.

Accepted by ApJ Supplements

References in corpus (5)

• Gravitational Waves and Gamma-rays from a Binary Neutron Star Merger: GW170817 and GRB 170817A
• An Ordinary Short Gamma-Ray Burst with Extraordinary Implications: Fermi-GBM Detection of GRB 170817A
• The 3rd Fermi GBM Gamma-Ray Burst Catalog: The First Six Years
• Evaluation of three methods for calculating statistical significance when incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process
• Significance in Gamma Ray Astronomy with Systematic Errors

Cited by in corpus (29)

• Gamma-ray bursts as cool synchrotron sources
• Gamma-ray Emission of 60Fe and 26Al Radioactivities in our Galaxy
• Bayesian Time-Resolved Spectroscopy of GRB Pulses
• Thermal Components in Gamma-ray Bursts. I. How Do They Affect Non-Thermal Spectral Parameters?
• The Fraction of Gamma-ray Bursts with an Observed Photospheric Emission Episode
• "Double-tracking" Characteristic of the Spectral Evolution of GRB 131231A: Synchrotron Origin?
• Estimate of the Background and Sensitivity of theFollow-up X-ray Telescope onboard Einstein Probe
• Non-dissipative photospheres in GRBs: Spectral appearance in the Fermi/GBM catalogue
• Classification of Photospheric Emission in Short GRBs
• Search for dark photons using a multilayer dielectric haloscope equipped with a single-photon avalanche diode
• On the $\alpha$-Intensity Correlation in Gamma-Ray Bursts: Subphotospheric Heating with Varying Entropy
• Testing High-latitude Curvature Effect of Gamma-Ray Bursts with {\it Fermi} Data: Evidence of Bulk Acceleration in Prompt Emission
• Periodicity Search on X-ray Bursts of SGR J1935+2154 Using 8.5 yr of Fermi/GBM Data
• Testing a model for subphotospheric dissipation in GRBs: fits to Fermi data constrain the dissipation scenario
• Relativistic accretion disk reflection in AGN X-ray spectra at z=0.5--4: a study of four \textit{Chandra} deep fields
• Prompt emission and early optical afterglow of VHE detected GRB 201015A and GRB 201216C: onset of the external forward shock
• Investigating subphotospheric dissipation in gamma-ray bursts using joint Fermi-Swift observations
• nazgul: A statistical approach to gamma-ray burst localization. Triangulation via non-stationary time-series models
• Searching for Axion-Like Particles from Core-Collapse Supernovae with Fermi LAT's Low Energy Technique
• One Fits All: A Unified Synchrotron Model Explains GRBs with FRED-Shaped Pulses
• Simulation Studies for the First Pathfinder of the CATCH Space Mission
• JCMT/SCUBA-2 uncovers an excess of $850\mu$m counts on megaparsec scales around high-redshift quasars. Characterization of the overdensities and their alignment with the quasars' Ly$\alpha$ nebulae
• Statistical Tools for Imaging Atmospheric Cherenkov Telescopes
• Evidence for a brief appearance of gamma-ray periodicity after a compact star merger
• Is magnetically dominated outflow required to explain GRBs?
• In-orbit background simulation of a type-B CATCH satellite
• Gamma-ray line emission from the Local Bubble
• Automatic detection of long-duration transients in Fermi-GBM data
• Time-averaging Polarimetric and Spectral Properties of Gamma-Ray Bursts