The Sample Complexity of Simple Binary Hypothesis Testing

Abstract

The sample complexity of simple binary hypothesis testing is the smallest number of i.i.d.\ samples required to distinguish between two distributions $p$ and $q$ in either: (i) the prior-free setting, with type-I error at most $\hat{I}\pm$ and type-II error at most ${\hat{I}}^2$; or (ii) the Bayesian setting, with Bayes error at most $\hat{I}´$ and prior distribution $(\ddot{I}€,1-\ddot{I}€)$. This problem has only been studied when $\hat{I}\pm ={\hat{I}}^2$ (prior-free) or $\ddot{I}€=1/2$ (Bayesian), and the sample complexity is known to be characterized by the Hellinger divergence between $p$ and $q$, up to multiplicative constants. In this paper, we derive a formula that characterizes the sample complexity (up to multiplicative constants that are independent of $p$, $q$, and all error parameters) for: (i) all $0\le \hat{I}\pm ,{\hat{I}}^2\le 1/8$ in the prior-free setting; and (ii) all $\hat{I}´\le \ddot{I}€/4$ in the Bayesian setting. In particular, the formula admits equivalent expressions in terms of certain divergences from the Jensen--Shannon and Hellinger families. The main technical result concerns an $f$-divergence inequality between members of the Jensen--Shannon and Hellinger families, which is proved by a combination of information-theoretic tools and case-by-case analyses. We explore applications of our results to (i) robust hypothesis testing, (ii) distributed (locally-private and communication-constrained) hypothesis testing, (iii) sequential hypothesis testing, and (iv) hypothesis testing with erasures.

Comments welcome. The new version includes results on sequential hypothesis testing and on hypothesis testing with erasures