The look-elsewhere effect from a unified Bayesian and frequentist perspective

Abstract

When searching over a large parameter space for anomalies such as events, peaks, objects, or particles, there is a large probability that spurious signals with seemingly high significance will be found. This is known as the look-elsewhere effect and is prevalent throughout cosmology, (astro)particle physics, and beyond. To avoid making false claims of detection, one must account for this effect when assigning the statistical significance of an anomaly. This is typically accomplished by considering the trials factor, which is generally computed numerically via potentially expensive simulations. In this paper we develop a continuous generalization of the Bonferroni and Sidak corrections by applying the Laplace approximation to evaluate the Bayes factor, and in turn relating the trials factor to the prior-to-posterior volume ratio. We use this to define a test statistic whose frequentist properties have a simple interpretation in terms of the global $p$-value, or statistical significance. We apply this method to various physics-based examples and show it to work well for the full range of $p$-values, i.e. in both the asymptotic and non-asymptotic regimes. We also show that this method naturally accounts for other model complexities such as additional degrees of freedom, generalizing Wilks' theorem. This provides a fast way to quantify statistical significance in light of the look-elsewhere effect, without resorting to expensive simulations.

21 pages, 7 figures

References in corpus (6)

• Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
• Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC
• Five-Year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Interpretation
• Measuring primordial non-gaussianity without cosmic variance
• Combining power spectrum and bispectrum measurements to detect oscillatory features
• Likelihood-Ratio Ranking Statistic for Compact Binary Coalescence Candidates with Rate Estimation

Cited by in corpus (15)

• Challenges for $\Lambda$CDM: An update
• Hints of Early Dark Energy in Planck, SPT, and ACT data: new physics or systematics?
• Colliding Ghosts: Constraining Inflation with the Parity-Odd Galaxy Four-Point Function
• Machine learning constraints on deviations from general relativity from the large scale structure of the Universe
• Primordial feature constraints from BOSS+eBOSS
• Detecting the periodicity of highly irregularly sampled light-curves with Gaussian processes: the case of SDSSJ025214.67-002813.7
• Matched filtering with non-Gaussian noise for planet transit detections
• On the absence of backsplash analogues to NGC 3109 in the $\Lambda$CDM framework
• pyBumpHunter: A model independent bump hunting tool in Python for High Energy Physics analyses
• Self-Calibrating the Look-Elsewhere Effect: Fast Evaluation of the Statistical Significance Using Peak Heights
• Statistical significance testing for mixed priors: a combined Bayesian and frequentist analysis
• Massive $\nu$s through the CNN lens: interpreting the field-level neutrino mass information in weak lensing
• Mechanical Sensors for Ultraheavy Dark Matter Searches via Long-range Forces
• Unification of Rare and Weak Detection Models using Moderate Deviations Analysis and Log-Chisquared P-values
• Fast estimation of the look-elsewhere effect using Gaussian random fields