Frequentist Evaluation of Intervals Estimated for a Binomial Parameter and for the Ratio of Poisson Means

Abstract

Confidence intervals for a binomial parameter or for the ratio of Poisson means are commonly desired in high energy physics (HEP) applications such as measuring a detection efficiency or branching ratio. Due to the discreteness of the data, in both of these problems the frequentist coverage probability unfortunately depends on the unknown parameter. Trade-offs among desiderata have led to numerous sets of intervals in the statistics literature, while in HEP one typically encounters only the classic intervals of Clopper-Pearson (central intervals with no undercoverage but substantial over-coverage) or a few approximate methods which perform rather poorly. If strict coverage is relaxed, some sort of averaging is needed to compare intervals. In most of the statistics literature, this averaging is over different values of the unknown parameter, which is conceptually problematic from the frequentist point of view in which the unknown parameter is typically fixed. In contrast, we perform an (unconditional) {\it average over observed data} in the ratio-of-Poisson-means problem. If strict conditional coverage is desired, we recommend Clopper-Pearson intervals and intervals from inverting the likelihood ratio test (for central and non-central intervals, respectively). Lancaster's mid-$P$ modification to either provides excellent unconditional average coverage in the ratio-of-Poisson-means problem.

V2 has clarifications in abstract, introduction, and conclusion. Published version has a subset of plots, is otherwise same as V2

References in corpus (3)

• Big Bang Nucleosynthesis (in "The Review of Particle Properties" 2004)
• Evaluation of three methods for calculating statistical significance when incorporating a systematic uncertainty into a test of the background-only hypothesis for a Poisson process
• Binomial and ratio-of-Poisson-means frequentist confidence intervals applied to the error evaluation of cut efficiencies

Cited by in corpus (8)

• On the Estimation of Confidence Intervals for Binomial Population Proportions in Astronomy: The Simplicity and Superiority of the Bayesian Approach
• Search for top squarks decaying to tau sleptons in $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector
• Simple and statistically sound recommendations for analysing physical theories
• Estimating the selection efficiency
• Lectures on Statistics in Theory: Prelude to Statistics in Practice
• Reference analysis of the signal + background model in counting experiments
• Search for resonant production of second-generation sleptons with same-sign dimuon events in proton-proton collisions at $\sqrt{s}=$ 13 TeV
• Statistical techniques to estimate the SARS-CoV-2 infection fatality rate