Testing One Hypothesis Multiple times

Abstract

In applied settings, tests of hypothesis where a nuisance parameter is only identifiable under the alternative often reduces into one of Testing One Hypothesis Multiple times (TOHM). Specifically, a fine discretization of the space of the non-identifiable parameter is specified, and the null hypothesis is tested against a set of sub-alternative hypothesis, one for each point of the discretization. The resulting sub-test statistics are then combined to obtain a global p-value. In this paper, we discuss a computationally efficient inferential tool to perform TOHM under stringent significance requirements, such as those typically required in the physical sciences, (e.g., p-value $<10^{-7}$). The resulting procedure leads to a generalized approach to perform inference under non-standard conditions, including non-nested models comparisons.

References in corpus (5)

• On hypothesis testing, trials factor, hypertests and the BumpHunter
• Random fields of multivariate test statistics, with applications to shape analysis
• Search for Gamma-Ray Lines towards Galaxy Clusters with the Fermi-LAT
• A method for comparing non-nested models with application to astrophysical searches for new physics
• Testing One Hypothesis Multiple Times: The Multidimensional Case

Cited by in corpus (2)

• Searching for new physics with profile likelihoods: Wilks and beyond
• Deficit hawks: robust new physics searches with unknown backgrounds