Axioms for testing with data-dependent levels and e-values

Abstract

The emerging literature on hypothesis testing with data-dependent and post-hoc significance levels relies on a particular extension of the Type-I error to data-dependent levels. Existing arguments for this extension are heuristic, and primarily motivated by a resulting connection to the e-value. Our first contribution is to show that it is uniquely characterized by three natural axioms. Our second contribution is to reverse the connection to the e-value, to show that three analogous axioms support a recently proposed decision-theoretic definition of the e-value. If one wishes to distinguish non-rejections at different decisions, we show three simple additional axioms suffice. Finally, we show that the relationship between e-values and post-hoc testing requires just two of these axioms.

The axiomatization is now much cleaner and no longer relies on extra assumptions on the notion of validity. The e-value contribution is now also more prominent