Entropy estimates of small data sets

Abstract

Estimating entropies from limited data series is known to be a non-trivial task. Naive estimations are plagued with both systematic (bias) and statistical errors. Here, we present a new 'balanced estimator' for entropy functionals Shannon, Rényi and Tsallis) specially devised to provide a compromise between low bias and small statistical errors, for short data series. This new estimator out-performs other currently available ones when the data sets are small and the probabilities of the possible outputs of the random variable are not close to zero. Otherwise, other well-known estimators remain a better choice. The potential range of applicability of this estimator is quite broad specially for biological and digital data series.

11 pages, 2 figures

References in corpus (8)

• Entropy estimation of symbol sequences
• Entropy and information in neural spike trains: Progress on the sampling problem
• Stochastic Models of Evolution in Genetics, Ecology and Linguistics
• Bias Analysis in Entropy Estimation
• Guessing probability distributions from small samples
• Guessing probability distributions from small samples
• Enlarged scaling ranges for the KS-entropy and the information dimension
• When do finite sample effects significantly affect entropy estimates ?

Cited by in corpus (15)

• JIDT: An information-theoretic toolkit for studying the dynamics of complex systems
• Mutual Information Rate-Based Networks in Financial Markets
• Partial Transfer Entropy on Rank Vectors
• Statistical Criticality arises in Most Informative Representations
• Hurst Exponents For Short Time Series
• Information-theoretic approach to lead-lag effect on financial markets
• Evaluation of Scale-Invariance In Physiological Signals By Means Of Balanced Estimation Of Diffusion Entropy
• Quantifying Relevance in Learning and Inference
• Non-equilibrium Dynamics, Thermalization and Entropy Production
• Entropy estimators for Markovian sequences: A comparative analysis
• Partial Mutual Information Analysis of Financial Networks
• Estimating entropy rate from censored symbolic time series: a test for time-irreversibility
• Mutual Information as a Two-Point Correlation Function in Stochastic Lattice Models
• Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability
• Experimentally accessible measurement of irreversibility in stochastic systems by categorizing single-molecule displacements