Inference of entropies of discrete random variables with unknown cardinalities

Abstract

We examine the recently introduced NSB estimator of entropies of severely undersampled discrete variables and devise a procedure for calculating the involved integrals. We discover that the output of the estimator has a well defined limit for large cardinalities of the variables being studied. Thus one can estimate entropies with no a priori assumptions about these cardinalities, and a closed form solution for such estimates is given.

8 pages

References in corpus (2)

• Entropy estimation of symbol sequences
• Entropy and information in neural spike trains: Progress on the sampling problem

Cited by in corpus (19)

• Entropy and information in neural spike trains: Progress on the sampling problem
• Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
• Maximum Likelihood Estimation of Functionals of Discrete Distributions
• Variational Gibbs State Preparation on NISQ devices
• Quantifying Relevance in Learning and Inference
• Estimating the Mutual Information between two Discrete, Asymmetric Variables with Limited Samples
• Limits on Inferring T-cell Specificity from Partial Information
• Pairwise Network Information and Nonlinear Correlations
• Estimating Functions of Distributions Defined over Spaces of Unknown Size
• Entropy estimators for Markovian sequences: A comparative analysis
• Emergence of functional information from multivariate correlations
• Low probability states, data statistics, and entropy estimation
• Neural coding of a natural stimulus ensemble: Uncovering information at sub-millisecond resolution
• Inferring a property of a large system from a small number of samples
• Chance, long tails, and inference: a non-Gaussian, Bayesian theory of vocal learning in songbirds
• Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability
• Nonparametric Maximum Entropy Estimation on Information Diagrams
• Millisecond-scale motor encoding in a cortical vocal area
• Bias-corrected estimator for intrinsic dimension and differential entropy--a visual multiscale approach