Equivalence testing with data-dependent and post-hoc equivalence margins

Abstract

Equivalence testing compares the hypothesis that an effect $\hat{I}¼$ is large against the alternative that it is negligible. Here, `large' is classically expressed as being larger than some `equivalence margin' $\hat{I}”$. A longstanding problem is that this margin must be specified but can rarely be objectively justified in practice. We lay the foundation for an alternative paradigm, arguing to instead report a data-dependent margin $\hat{\hat{I}}{”}_{\hat{I}}\pm$ that bounds the true effect $\hat{I}¼$ with probability $1-\hat{I}\pm$. Our key argument is that $\hat{\hat{I}}{”}_{\hat{I}}\pm$ is more useful than a test outcome at a fixed margin $\hat{I}”$, as measured by the guarantees it offers to decision makers. We generalize this to a curve of margins $\hat{I}\pm \mapsto \hat{\hat{I}}{”}_{\hat{I}}\pm$, uniformly valid under the post-hoc selection of the margin. These ideas rely on e-values, which we derive for models that are strictly totally positive of order 3, nesting the classical z-test and t-test settings.