Probing high order dependencies with information theory

Abstract

Information theoretic measures (entropies, entropy rates, mutual information) are nowadays commonly used in statistical signal processing for real-world data analysis. The present work proposes the use of Auto Mutual Information (Mutual Information between subsets of the same signal) and entropy rate as powerful tools to assess refined dependencies of any order in signal temporal dynamics. Notably, it is shown how two-point Auto Mutual Information and entropy rate unveil information conveyed by higher order statistic and thus capture details of temporal dynamics that are overlooked by the (two-point) correlation function. Statistical performance of relevant estimators for Auto Mutual Information and entropy rate are studied numerically, by means of Monte Carlo simulations, as functions of sample size, dependence structures and hyper parameters that enter their definition. Further, it is shown how Auto Mutual Information permits to discriminate between several different non Gaussian processes, having exactly the same marginal distribution and covariance function. Assessing higher order statistics via multipoint Auto Mutual Information is also shown to unveil the global dependence structure fo these processes, indicating that one of the non Gaussian actually has temporal dynamics that ressembles that of a Gaussian process with same covariance while the other does not.

References in corpus (13)

• Estimating Mutual Information
• Power-law Strength-Degree Correlation From a Resource-Allocation Dynamics on Weighted Networks
• The Three-Point Correlation Function in Cosmology
• The relation between Granger causality and directed information theory: a review
• On directed information theory and Granger causality graphs
• Computing the Three-Point Correlation Function of Galaxies in $\mathcal{O}(N^2)$ Time
• Analytic Minkowski Functionals of the Cosmic Microwave Background: Second-order Non-Gaussianity with Bispectrum and Trispectrum
• Cosmic Microwave Background Trispectrum and Primordial Magnetic Field Limits
• Estimation of time-delayed mutual information and bias for irregularly and sparsely sampled time-series
• Using time-delayed mutual information to discover and interpret temporal correlation structure in complex populations
• Mutual Information in Frequency and its Application to Measure Cross-Frequency Coupling in Epilepsy
• Scaling of Information in Turbulence
• Information Theory to probe Intrapartum Fetal Heart Rate Dynamics

Cited by in corpus (5)

• Information theory for non-stationary processes with stationary increments
• Distance to healthy cardiovascular dynamics from fetal heart rate scale-dependent features in pregnant sheep model of human labor predicts cardiovascular decompensation
• Quantifying Non-Stationarity with Information Theory
• A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields
• Explicit mutual information for simple networks and neurons with lognormal activities