Estimating the entropy of binary time series: Methodology, some theory and a simulation study

Abstract

Partly motivated by entropy-estimation problems in neuroscience, we present a detailed and extensive comparison between some of the most popular and effective entropy estimation methods used in practice: The plug-in method, four different estimators based on the Lempel-Ziv (LZ) family of data compression algorithms, an estimator based on the Context-Tree Weighting (CTW) method, and the renewal entropy estimator. **Methodology. Three new entropy estimators are introduced. For two of the four LZ-based estimators, a bootstrap procedure is described for evaluating their standard error, and a practical rule of thumb is heuristically derived for selecting the values of their parameters. ** Theory. We prove that, unlike their earlier versions, the two new LZ-based estimators are consistent for every finite-valued, stationary and ergodic process. An effective method is derived for the accurate approximation of the entropy rate of a finite-state HMM with known distribution. Heuristic calculations are presented and approximate formulas are derived for evaluating the bias and the standard error of each estimator. ** Simulation. All estimators are applied to a wide range of data generated by numerous different processes with varying degrees of dependence and memory. Some conclusions drawn from these experiments include: (i) For all estimators considered, the main source of error is the bias. (ii) The CTW method is repeatedly and consistently seen to provide the most accurate results. (iii) The performance of the LZ-based estimators is often comparable to that of the plug-in method. (iv) The main drawback of the plug-in method is its computational inefficiency.

34 pages, 3 figures

Cited by in corpus (15)

• Mutual Information Rate-Based Networks in Financial Markets
• Complex systems approach to natural language
• Gibbs distribution analysis of temporal correlations structure in retina ganglion cells
• A view of Neural Networks as dynamical systems
• Path Weight Sampling: Exact Monte Carlo Computation of the Mutual Information between Stochastic Trajectories
• On the emergence of Zipf's law in music
• A hybrid analysis of LBSN data to early detect anomalies in crowd dynamics
• Posterior Representations for Bayesian Context Trees: Sampling, Estimation and Convergence
• How Gibbs distributions may naturally arise from synaptic adaptation mechanisms. A model-based argumentation
• Recurrence times, waiting times and universal entropy production estimators
• Maximal Repetition and Zero Entropy Rate
• Time-irreversibility tests for random-length time series: the matching-time approach applied to DNA
• Entropic analysis of the localization-delocalization transition in a one-dimensional correlated lattice
• Non-parametric change-point detection using string matching algorithms
• Repetition and recurrence times: Dual statements and summable mixing rates