Chebyshev polynomials, moment matching, and optimal estimation of the unseen

Abstract

We consider the problem of estimating the support size of a discrete distribution whose minimum non-zero mass is at least $\frac{1}{k}$. Under the independent sampling model, we show that the sample complexity, i.e., the minimal sample size to achieve an additive error of $\epsilon k$ with probability at least 0.1 is within universal constant factors of $\frac{k}{\log k}{\log }^2\frac{1}{\epsilon }$, which improves the state-of-the-art result of $\frac{k}{{\epsilon }^2\log k}$ in \cite{VV13}. Similar characterization of the minimax risk is also obtained. Our procedure is a linear estimator based on the Chebyshev polynomial and its approximation-theoretic properties, which can be evaluated in $O(n+{\log }^{2}k)$ time and attains the sample complexity within a factor of six asymptotically. The superiority of the proposed estimator in terms of accuracy, computational efficiency and scalability is demonstrated in a variety of synthetic and real datasets.

References in corpus (2)

• Testing composite hypotheses, Hermite polynomials and optimal estimation of a nonsmooth functional
• Estimating the number of classes

Cited by in corpus (11)

• Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
• Measuring Quantum Entropy
• Optimal estimation of Gaussian mixtures via denoised method of moments
• Approximate Profile Maximum Likelihood
• Maximum Likelihood Estimation for Learning Populations of Parameters
• Adaptive Estimation of Shannon Entropy
• Quantum query complexity of entropy estimation
• Estimating the number of unseen species: A bird in the hand is worth $\log n$ in the bush
• Minimax Optimal Additive Functional Estimation with Discrete Distribution
• Testing Properties of Multiple Distributions with Few Samples
• Towards Testing Monotonicity of Distributions Over General Posets