Axioms for validity: e-values, p-values and data-dependent levels

Abstract

The emerging literature on hypothesis testing with data-dependent significance levels relies on a particular extension of the Type-I error. Existing arguments for this extension are heuristic. We show it is uniquely characterized by three axioms: law-invariance, agreement with classical testing on constant levels, and the preservation of (in)validity under independent selection of the level. We find that three analogous axioms characterize the e-value as a multi-decision generalization of a test in a decision-theoretic framework. We recover the p-value by dropping the invalidity half of the independent selection axiom, showing that e-values correspond to those p-values for which a data-independent choice between two invalid p-values cannot produce a valid p-value. Our main technical contribution is to show how e-values emerge in every decision problem with a notion of validity that satisfies law invariance, the independent selection axiom, and an axiom stating that a procedure that selects an invalid procedure with arbitrarily high probability must itself be invalid. In an application to post-hoc decision-making, we justify expectation-based error rates in multiple testing and recover the 'e-closure principle' as a special case.

arXiv commentGeneralized Section 5 to arbitrary decision spaces, now recovering calibration to classical testing instead of imposing it. Generalized the post-hoc section to arbitrary decision problems, recovering the e-closure principle. Also improved the distinguishing non-rejections argumentation