Entropy estimators for Markovian sequences: A comparative analysis

Abstract

Entropy estimation is a fundamental problem in information theory that has applications in various fields, including physics, biology, and computer science. Estimating the entropy of discrete sequences can be challenging due to limited data and the lack of unbiased estimators. Most existing entropy estimators are designed for sequences of independent events and their performance vary depending on the system being studied and the available data size. In this work we compare different entropy estimators and their performance when applied to Markovian sequences. Specifically, we analyze both binary Markovian sequences and Markovian systems in the undersampled regime. We calculate the bias, standard deviation and mean squared error for some of the most widely employed estimators. We discuss the limitations of entropy estimation as a function of the transition probabilities of the Markov processes and the sample size. Overall, this paper provides a comprehensive comparison of entropy estimators and their performance in estimating entropy for systems with memory, which can be useful for researchers and practitioners in various fields.

19 pages, 9 figures

References in corpus (5)

• Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks
• Entropy estimates of small data sets
• Inferring Markov Chains: Bayesian Estimation, Model Comparison, Entropy Rate, and Out-of-class Modeling
• Regular and stochastic behavior of Parkinsonian pathological tremor signals
• Tackling the subsampling problem to infer collective properties from limited data

Cited by in corpus (4)

• infomeasure: A Comprehensive Python Package for Information Theory Measures and Estimators
• Entropy and type-token ratio in gigaword corpora
• Exploring language relations through syntactic distances and geographic proximity
• Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability