Near-Optimal Closeness Testing of Discrete Histogram Distributions

Abstract

We investigate the problem of testing the equivalence between two discrete histograms. A {\em $k$-histogram} over $[n]$ is a probability distribution that is piecewise constant over some set of $k$ intervals over $[n]$. Histograms have been extensively studied in computer science and statistics. Given a set of samples from two $k$-histogram distributions $p,q$ over $[n]$, we want to distinguish (with high probability) between the cases that $p=q$ and $\Vert p-q{\Vert }_1\ge \epsilon$. The main contribution of this paper is a new algorithm for this testing problem and a nearly matching information-theoretic lower bound. Specifically, the sample complexity of our algorithm matches our lower bound up to a logarithmic factor, improving on previous work by polynomial factors in the relevant parameters. Our algorithmic approach applies in a more general setting and yields improved sample upper bounds for testing closeness of other structured distributions as well.

References in corpus (3)

• Learning mixtures of structured distributions over discrete domains
• Fast Algorithms for Segmented Regression
• Near-Optimal Density Estimation in Near-Linear Time Using Variable-Width Histograms

Cited by in corpus (5)

• Differentially Private Identity and Closeness Testing of Discrete Distributions
• Sharp Bounds for Generalized Uniformity Testing
• Sequential Change Detection by Optimal Weighted ${\ell }_2$ Divergence
• Unfolding by Folding: a resampling approach to the problem of matrix inversion without actually inverting any matrix
• Goodness-of-Fit Testing for Hölder-Continuous Densities: Sharp Local Minimax Rates