Entropy Estimates from Insufficient Samplings
arXiv:physics/0307138
Abstract
We present a detailed derivation of some estimators of Shannon entropy for discrete distributions. They hold for finite samples of N points distributed into M "boxes", with N and M -> oo, but N/M < oo. In the high sampling regime (<< 1 points in each box) they have exponentially small biases. In the low sampling regime the errors increase but are still much smaller than for most other estimators. One advantage is that our main estimators are given analytically, with explicitly known analytical formulas for the biases.
5 pages, including 2 figures; revised version with corrected figures
Cited by in corpus (18)
- Maximum Likelihood Estimation of Functionals of Discrete Distributions
- Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
- Partial Transfer Entropy on Rank Vectors
- Offdiagonal Complexity: A computationally quick complexity measure for graphs and networks
- Production and characterization of a fragmented spinor Bose-Einstein condensate
- Improved Information Gain Estimates for Decision Tree Induction
- High-Order Conditional Mutual Information Maximization for dealing with High-Order Dependencies in Feature Selection
- Causation entropy from symbolic representations of dynamical systems
- Dynamical response of the Hodgkin-Huxley model in the high-input regime
- A note on entropy estimation
- Estimating Functions of Distributions Defined over Spaces of Unknown Size
- Understanding Recurrent Neural Architectures by Analyzing and Synthesizing Long Distance Dependencies in Benchmark Sequential Datasets
- Entropy from Machine Learning
- Mutual Information Scaling and Expressive Power of Sequence Models
- MIC: Mutual Information based hierarchical Clustering
- Inferring a property of a large system from a small number of samples
- Monte Carlo convergence of rival samplers
- Entropy and efficiency of the ETF market