Minimax Rate-Optimal Estimation of Divergences between Discrete Distributions

Abstract

We study the minimax estimation of $\hat{I}\pm$-divergences between discrete distributions for integer $\hat{I}\pm \ge 1$, which include the Kullback--Leibler divergence and the $\ddot{I}{‡}^2$-divergences as special examples. Dropping the usual theoretical tricks to acquire independence, we construct the first minimax rate-optimal estimator which does not require any Poissonization, sample splitting, or explicit construction of approximating polynomials. The estimator uses a hybrid approach which solves a problem-independent linear program based on moment matching in the non-smooth regime, and applies a problem-dependent bias-corrected plug-in estimator in the smooth regime, with a soft decision boundary between these regimes.

This (v5) is a significantly revised version of (v2), and fixed some typos in (v4)