Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks

Abstract

We present a procedure for effective estimation of entropy and mutual information from small-sample data, and apply it to the problem of inferring high-dimensional gene association networks. Specifically, we develop a James-Stein-type shrinkage estimator, resulting in a procedure that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperforms eight other entropy estimation procedures across a diverse range of sampling scenarios and data-generating models, even in cases of severe undersampling. We illustrate the approach by analyzing E. coli gene expression data and computing an entropy-based gene-association network from gene expression data. A computer program is available that implements the proposed shrinkage estimator.

18 pages, 3 figures, 1 table

References in corpus (2)

• Estimating high-dimensional directed acyclic graphs with the PC-algorithm
• Entropy estimation of symbol sequences

Cited by in corpus (25)

• Equitability, mutual information, and the maximal information coefficient
• Feature selection in omics prediction problems using cat scores and false nondiscovery rate control
• Maximum Likelihood Estimation of Functionals of Discrete Distributions
• Does Dirichlet Prior Smoothing Solve the Shannon Entropy Estimation Problem?
• Ideological and Temporal Components of Network Polarization in Online Political Participatory Media
• Differential protein expression and peak selection in mass spectrometry data by binary discriminant analysis
• Kernel Mean Shrinkage Estimators
• Hierarchical Multinomial-Dirichlet model for the estimation of conditional probability tables
• Bayesian Network Structure Learning with Permutation Tests
• The word entropy of natural languages
• Entropy estimators for Markovian sequences: A comparative analysis
• Entropy from Machine Learning
• Bayesian Entropy Estimation for Countable Discrete Distributions
• Profile Entropy: A Fundamental Measure for the Learnability and Compressibility of Discrete Distributions
• EEG Spatial Decoding and Classification with Logit Shrinkage Regularized Directed Information Assessment (L-SODA)
• Memory Lens: How Much Memory Does an Agent Use?
• Graph estimation with joint additive models
• ComplexityMeasures.jl: scalable software to unify and accelerate entropy and complexity timeseries analysis
• Information-theoretic signatures of biodiversity in the barcoding gene
• Measures of Variability for Bayesian Network Graphical Structures
• Structural Biases for Improving Transformers on Translation into Morphologically Rich Languages
• On Identifying Significant Edges in Graphical Models of Molecular Networks
• Fast calculation of entropy with Zhang's estimator
• Ranking Biomarkers Through Mutual Information
• Adversarial Orthogonal Regression: Two non-Linear Regressions for Causal Inference